Kubernetes PKI, kubeconfig Users, CSR and Certificate Renewal

kubeadm stores the cluster trust material under /etc/kubernetes/pki and embeds client identities in kubeconfig files. This lesson maps those files, creates a new user through a CertificateSigningRequest, and renews the certificates kubeadm manages before they expire.

Map the kubeadm PKI directory

On a control-plane node, list the certificate and key files kubeadm created:

sudo find /etc/kubernetes/pki -maxdepth 2 -type f | sort

Sample output:

/etc/kubernetes/pki/apiserver.crt
/etc/kubernetes/pki/apiserver-etcd-client.crt
/etc/kubernetes/pki/apiserver-etcd-client.key
/etc/kubernetes/pki/apiserver.key
/etc/kubernetes/pki/apiserver-kubelet-client.crt
/etc/kubernetes/pki/apiserver-kubelet-client.key
/etc/kubernetes/pki/ca.crt
/etc/kubernetes/pki/ca.key
/etc/kubernetes/pki/etcd/ca.crt
/etc/kubernetes/pki/etcd/ca.key
/etc/kubernetes/pki/etcd/healthcheck-client.crt
/etc/kubernetes/pki/etcd/healthcheck-client.key
/etc/kubernetes/pki/etcd/peer.crt
/etc/kubernetes/pki/etcd/peer.key
/etc/kubernetes/pki/etcd/server.crt
/etc/kubernetes/pki/etcd/server.key
/etc/kubernetes/pki/front-proxy-ca.crt
/etc/kubernetes/pki/front-proxy-ca.key
/etc/kubernetes/pki/front-proxy-client.crt
/etc/kubernetes/pki/front-proxy-client.key
/etc/kubernetes/pki/sa.key
/etc/kubernetes/pki/sa.pub

Group the files by the connection they protect:

Inspect the API serving certificate SANs so you know which names and IPs clients may use:

sudo openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -subject -issuer -dates -ext subjectAltName

Sample output:

subject=CN=kube-apiserver
issuer=CN=kubernetes
notBefore=Jul 24 08:19:23 2026 GMT
notAfter=Jul 24 08:24:23 2027 GMT
X509v3 Subject Alternative Name:
    DNS:k8s-cp, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, IP Address:10.96.0.1, IP Address:192.168.56.108

The issuer CN=kubernetes is the cluster CA. A missing VIP or DNS name in the SAN list is a common join and kubectl trust failure when you later add a load balancer.

Understand certificates inside kubeconfig files

kubeadm also writes kubeconfig files under /etc/kubernetes/. Each file is a small cluster identity package: which API to call, which CA to trust, and which client certificate to present.

sudo ls -la /etc/kubernetes/*.conf

Sample output:

-rw-------. 1 root root 5661 Jul 27 08:47 /etc/kubernetes/admin.conf
-rw-------. 1 root root 5662 Jul 24 13:54 /etc/kubernetes/controller-manager.conf
-rw-------. 1 root root 1954 Jul 24 13:54 /etc/kubernetes/kubelet.conf
-rw-------. 1 root root 5614 Jul 24 13:54 /etc/kubernetes/scheduler.conf
-rw-------. 1 root root 5666 Jul 24 13:54 /etc/kubernetes/super-admin.conf

Every kubeconfig has the same four ideas:

• Cluster — API server URL and certificate-authority-data (or a CA file path)
• User — client certificate and key (embedded data, or paths for kubelet)
• Context — binds one cluster, one user, and an optional default namespace
• current-context — which binding kubectl uses

Decode the embedded client identity for admin.conf:

sudo kubectl config view --raw --minify \
  --kubeconfig=/etc/kubernetes/admin.conf \
  -o jsonpath='{.users[0].user.client-certificate-data}' |
base64 -d |
openssl x509 -noout -subject

Sample output:

subject=O=kubeadm:cluster-admins, CN=kubernetes-admin

super-admin.conf is normally generated only on the node where kubeadm init ran, not on control-plane nodes added with kubeadm join --control-plane. Check before you decode it:

if sudo test -f /etc/kubernetes/super-admin.conf; then
  sudo kubectl config view --raw --minify \
    --kubeconfig=/etc/kubernetes/super-admin.conf \
    -o jsonpath='{.users[0].user.client-certificate-data}' |
  base64 -d |
  openssl x509 -noout -subject
else
  echo "super-admin.conf is not present on this control-plane node"
fi

Sample output on the init node:

subject=O=system:masters, CN=kubernetes-super-admin

On current kubeadm:

• admin.conf — day-to-day cluster-admin kubeconfig (kubernetes-admin in kubeadm:cluster-admins)
• super-admin.conf — break-glass system:masters access if RBAC bindings for the admin group are broken
• Prefer admin.conf for normal work; keep super-admin.conf root-readable only on the control plane
• ls, kubeadm certs check-expiration, and kubeadm certs renew all can omit super-admin.conf on joined control-plane nodes

kubelet.conf often points at rotating files under /var/lib/kubelet/pki/ instead of embedding the leaf certificate. That is expected—kubelet rotates its client certificate independently of kubeadm certs renew.

Inspect certificate subjects and expiry

Use kubeadm's summary first. It reports every certificate kubeadm knows about and whether the signing CA is externally managed:

sudo kubeadm certs check-expiration

Sample output (trimmed):

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Jul 24, 2027 08:24 UTC   361d            ca                      no
apiserver                  Jul 24, 2027 08:24 UTC   361d            ca                      no
apiserver-etcd-client      Jul 24, 2027 08:24 UTC   361d            etcd-ca                 no
apiserver-kubelet-client   Jul 24, 2027 08:24 UTC   361d            ca                      no
controller-manager.conf    Jul 24, 2027 08:24 UTC   361d            ca                      no
etcd-healthcheck-client    Jul 24, 2027 08:24 UTC   361d            etcd-ca                 no
etcd-peer                  Jul 24, 2027 08:24 UTC   361d            etcd-ca                 no
etcd-server                Jul 24, 2027 08:24 UTC   361d            etcd-ca                 no
front-proxy-client         Jul 24, 2027 08:24 UTC   361d            front-proxy-ca          no
scheduler.conf             Jul 24, 2027 08:24 UTC   361d            ca                      no
super-admin.conf           Jul 24, 2027 08:24 UTC   361d            ca                      no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Jul 21, 2036 08:24 UTC   9y              no
etcd-ca                 Jul 21, 2036 08:24 UTC   9y              no
front-proxy-ca          Jul 21, 2036 08:24 UTC   9y              no

By default, kubeadm certificate lifetimes work like this:

• Non-CA certificates — one year
• CA certificates — ten years
• kubeadm v1beta4 — change either period through certificateValidityPeriod and caCertificateValidityPeriod
• These are defaults, not fixed Kubernetes limits
• EXTERNALLY MANAGED stays no when the CA private key lives in /etc/kubernetes/pki
• kubeadm does not track custom certificates you add yourself (webhook TLS, Ingress secrets, and so on)

Zoom in on any file with OpenSSL when you need subject or SAN detail:

sudo openssl x509 -in /etc/kubernetes/pki/ca.crt -noout -subject -issuer -dates

Sample output:

subject=CN=kubernetes
issuer=CN=kubernetes
notBefore=Jul 24 08:19:23 2026 GMT
notAfter=Jul 21 08:24:23 2036 GMT

Create a Kubernetes client certificate request

A client certificate is how a human or automation proves identity to the API server:

• Certificate CN becomes the API username
• Organizations (O=) become groups
• Those strings must match what you later bind in RBAC — authentication alone grants nothing

Generate a key and OpenSSL CSR for user alice in group pki-lab-users:

mkdir -p "$HOME/pki-lab"

cd "$HOME/pki-lab"

openssl genrsa -out alice.key 2048

openssl req -new -key alice.key -out alice.csr -subj "/CN=alice/O=pki-lab-users"

Those commands write alice.key and alice.csr with no terminal output on success.

CertificateSigningRequest.spec is immutable. Delete any prior alice object before you apply a new CSR so the workflow stays rerunnable:

kubectl delete csr alice --ignore-not-found

No output is shown when the CSR does not already exist.

Create a Kubernetes CertificateSigningRequest that asks the built-in API client signer to issue a client-auth certificate. Creating and approving CSRs need different API access:

• Create — access to certificatesigningrequests
• Approve — update the approval subresource and permission to approve the selected signer

cat <<EOF | kubectl apply -f -
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: alice
spec:
  request: $(base64 -w0 alice.csr)
  signerName: kubernetes.io/kube-apiserver-client
  expirationSeconds: 86400
  usages:
  - client auth
EOF

Sample output:

certificatesigningrequest.certificates.k8s.io/alice created

kubernetes.io/kube-apiserver-client is the signer for API client certificates. usages: [client auth] matches that purpose. Do not put cluster-admin groups into the CSR unless you intend that identity to match powerful RBAC subjects.

List the CSR while it waits for approval:

kubectl get csr alice

Sample output:

NAME    AGE   SIGNERNAME                            REQUESTOR          REQUESTEDDURATION   CONDITION
alice   1s    kubernetes.io/kube-apiserver-client   kubernetes-admin   24h                 Pending

Pending means the object exists but is not yet approved. Describe it to confirm the subject before you approve:

kubectl describe csr alice

Sample output (trimmed):

Name:         alice
Signer:       kubernetes.io/kube-apiserver-client
Requested Duration:  24h
Status:              Pending
Subject:
         Common Name:    alice
         Organization:   pki-lab-users

Approve or deny a CSR

Approval is a separate API decision from signing. For the built-in signer, approving a valid request also triggers issuance.

Approve Alice's request:

kubectl certificate approve alice

Sample output:

certificatesigningrequest.certificates.k8s.io/alice approved

Approval and signing are asynchronous. Wait until the signed certificate appears in .status.certificate before you extract it:

kubectl wait \
  --for=jsonpath='{.status.certificate}' \
  csr/alice --timeout=30s

Sample output:

certificatesigningrequest.certificates.k8s.io/alice condition met

kubectl get csr alice

Sample output:

NAME    AGE   SIGNERNAME                            REQUESTOR          REQUESTEDDURATION   CONDITION
alice   2s    kubernetes.io/kube-apiserver-client   kubernetes-admin   24h                 Approved,Issued

Approved,Issued means the certificate bytes are in .status.certificate.

To refuse a request instead:

• Run kubectl certificate deny <csr-name> — that adds the Denied condition
• The signer must not issue a certificate for that request
• Delete denied CSRs when you are done experimenting

Retrieve the issued certificate

Extract the PEM certificate from the CSR status and verify it:

kubectl get csr alice -o jsonpath='{.status.certificate}' | base64 -d > alice.crt

openssl x509 -in alice.crt -noout -subject -issuer -dates -ext extendedKeyUsage

Sample output:

subject=O=pki-lab-users, CN=alice
issuer=CN=kubernetes
notBefore=Jul 27 12:23:38 2026 GMT
notAfter=Jul 28 12:23:38 2026 GMT
X509v3 Extended Key Usage:
    TLS Web Client Authentication

What to check in the issued certificate:

• Issuer should be the cluster CA; subject should match the CSR
• Extended key usage should be client authentication
• The CSR requests 24 hours, but the actual lifetime is the shorter of the requested duration and the signer's configured maximum
• Verify notBefore and notAfter — the signer is permitted to issue a different duration

Build a kubeconfig for the new user

Point a dedicated kubeconfig at the cluster CA, Alice's certificate, and her key. Extract the API endpoint and CA from your currently working kubeconfig so the file works on HA clusters and other installations:

API_SERVER=$(kubectl config view --raw --minify \
  -o jsonpath='{.clusters[0].cluster.server}')

kubectl config view --raw --minify --flatten \
  -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' |
base64 -d > cluster-ca.crt

kubectl config --kubeconfig=alice.kubeconfig set-cluster kubernetes \
  --server="$API_SERVER" \
  --certificate-authority=cluster-ca.crt \
  --embed-certs=true

kubectl config --kubeconfig=alice.kubeconfig set-credentials alice \
  --client-certificate=alice.crt \
  --client-key=alice.key \
  --embed-certs=true

kubectl config --kubeconfig=alice.kubeconfig set-context alice@kubernetes \
  --cluster=kubernetes \
  --user=alice \
  --namespace=default

kubectl config --kubeconfig=alice.kubeconfig use-context alice@kubernetes

Each set-* command prints a short confirmation. Confirm authentication works:

kubectl --kubeconfig=alice.kubeconfig auth whoami

Sample output:

ATTRIBUTE                                           VALUE
Username                                            alice
Groups                                              [pki-lab-users system:authenticated]
Extra: authentication.kubernetes.io/credential-id   [X509SHA256=c138c7ab8e352b44982463d5892eec51c3404adfdb81e2461be9e43e461fbe99]

The API accepted the certificate. Authorization is still empty until you bind RBAC. Prove that:

kubectl --kubeconfig=alice.kubeconfig get pods -A

Sample output:

Error from server (Forbidden): pods is forbidden: User "alice" cannot list resource "pods" in API group "" at the cluster scope

kubectl --kubeconfig=alice.kubeconfig auth can-i get pods --all-namespaces

Sample output:

no

Authentication succeeded; authorization did not. Create RoleBindings or ClusterRoleBindings that match alice or group pki-lab-users—see Kubernetes RBAC with examples.

Renew kubeadm-managed certificates

When residual time in kubeadm certs check-expiration gets short, renew on each control-plane node:

• kubeadm upgrade apply and kubeadm upgrade node renew kubeadm-managed certificates on that control-plane node by default
• Manual renewal is mainly required when certificates must be renewed independently of an upgrade, or upgrades will not happen within the certificate lifetime
• Automatic renewal during kubeadm upgrades is enabled unless --certificate-renewal=false is supplied

Renewal is unconditional: kubeadm reissues even if the current certificate is still valid, and it reuses attributes such as SANs from the existing files.

Renew everything kubeadm manages:

sudo kubeadm certs renew all

Sample output:

certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed
certificate embedded in the kubeconfig file for the super-admin renewed

Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.

For a controlled change, renew only the administrator kubeconfig:

sudo kubeadm certs renew admin.conf

Sample output:

certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed

After renewal, what you do next depends on scope:

• admin.conf only — skip the static-Pod restart and refresh $HOME/.kube/config
• renew all or component certificates — restart the control-plane static Pods that consume those files

After renew all, confirm residual time moved forward:

sudo kubeadm certs check-expiration

Sample output (trimmed):

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Jul 27, 2027 12:29 UTC   364d            ca                      no
apiserver                  Jul 27, 2027 12:29 UTC   364d            ca                      no
apiserver-etcd-client      Jul 27, 2027 12:29 UTC   364d            etcd-ca                 no
apiserver-kubelet-client   Jul 27, 2027 12:29 UTC   364d            ca                      no
controller-manager.conf    Jul 27, 2027 12:29 UTC   364d            ca                      no
etcd-healthcheck-client    Jul 27, 2027 12:29 UTC   364d            etcd-ca                 no
etcd-peer                  Jul 27, 2027 12:29 UTC   364d            etcd-ca                 no
etcd-server                Jul 27, 2027 12:29 UTC   364d            etcd-ca                 no
front-proxy-client         Jul 27, 2027 12:29 UTC   364d            front-proxy-ca          no
scheduler.conf             Jul 27, 2027 12:29 UTC   364d            ca                      no
super-admin.conf           Jul 27, 2027 12:29 UTC   364d            ca                      no

New files on disk do not take effect until control-plane processes reload them:

• kubelet polls /etc/kubernetes/manifests on an interval
• Wait for kubelet to terminate the static Pod after manifest removal
• Wait again after restoring the manifest
• Capture the original container ID, wait until it stops, then wait for a different ID

The helper below implements that sequence:

restart_static_pod() {
  component="$1"
  manifest="/etc/kubernetes/manifests/${component}.yaml"
  temporary="/tmp/${component}.yaml.off"

  old_id=$(sudo crictl ps \
    --state Running \
    --name "^${component}$" \
    -q | head -n1)

  sudo mv "$manifest" "$temporary"

  while sudo crictl ps \
    --state Running \
    --name "^${component}$" \
    -q | grep -q .; do
    sleep 2
  done

  sudo mv "$temporary" "$manifest"

  while true; do
    new_id=$(sudo crictl ps \
      --state Running \
      --name "^${component}$" \
      -q | head -n1)

    if [ -n "$new_id" ] && [ "$new_id" != "$old_id" ]; then
      break
    fi

    sleep 2
  done

  echo "$component restarted"
}

Run one component at a time:

restart_static_pod etcd

restart_static_pod kube-apiserver

until curl -sk https://127.0.0.1:6443/readyz |
  grep -qx 'ok'; do
  sleep 2
done

Confirm the running API server loaded the renewed serving certificate. kubeadm certs check-expiration reads files on disk; it does not prove the process in memory picked up the new cert:

sudo openssl x509 \
  -in /etc/kubernetes/pki/apiserver.crt \
  -noout -serial -enddate

echo | openssl s_client \
  -connect 127.0.0.1:6443 2>/dev/null |
openssl x509 -noout -serial -enddate

Sample output:

serial=<renewed-certificate-serial>
notAfter=<renewed-expiry-date>

serial=<renewed-certificate-serial>
notAfter=<renewed-expiry-date>

What the comparison proves:

• Serial number and expiry from the live API endpoint should match /etc/kubernetes/pki/apiserver.crt
• If they differ, the API server is still serving the previous certificate
• Static Pods must be restarted because not every component dynamically reloads renewed certificates

restart_static_pod kube-controller-manager

restart_static_pod kube-scheduler

Sample output:

etcd restarted
kube-apiserver restarted
kube-controller-manager restarted
kube-scheduler restarted

On an HA control plane, complete and verify all four components on one control-plane node before renewing the next node—see high availability with kubeadm.

Refresh administrator kubeconfig copies

kubeadm certs renew admin.conf rewrites /etc/kubernetes/admin.conf. It does not update $HOME/.kube/config if that file is an older copy.

Compare embedded end dates:

sudo kubectl config view --raw --minify \
  --kubeconfig=/etc/kubernetes/admin.conf \
  -o jsonpath='{.users[0].user.client-certificate-data}' |
base64 -d |
openssl x509 -noout -enddate

Sample output after renewal:

notAfter=Jul 27 12:28:55 2027 GMT

openssl x509 -in <(kubectl config view --raw --minify --kubeconfig=$HOME/.kube/config -o jsonpath='{.users[0].user.client-certificate-data}' | base64 -d) -noout -enddate

Sample output when the home copy was still stale:

notAfter=Jul 24 08:24:23 2027 GMT

Refresh the home copy from the renewed file:

mkdir -p "$HOME/.kube"

sudo cp -f /etc/kubernetes/admin.conf "$HOME/.kube/config"

sudo chown "$(id -u):$(id -g)" "$HOME/.kube/config"

chmod 600 "$HOME/.kube/config"

Confirm kubectl still works:

kubectl get nodes

Sample output:

NAME       STATUS   ROLES           AGE    VERSION
k8s-cp     Ready    control-plane   3d4h   v1.36.3
worker01   Ready    <none>          168m   v1.36.3

Distribute renewed kubeconfigs to workstations the same way—file copy, not automatic sync.

Diagnose certificate expiry and trust errors

Check node time before you chase certificate bugs. Drift can make a valid certificate look expired or not yet valid.

kubeadm uses the existing certificate as the source for renewed CN, organization, and SAN values, and does not support CA replacement automatically.

External CA boundary

kubeadm can run when the cluster CA private key is not present on the node (external CA mode). In that layout:

• Leaf certificates are still expected under /etc/kubernetes/pki
• kubeadm certs renew cannot sign new leaves without the CA key
• kubeadm certs generate-csr writes CSR and key pairs (and partial kubeconfigs) for you to submit to the external CA

sudo kubeadm certs generate-csr --help

The help text states the command is designed for kubeadm external CA mode and that you save signed PEMs beside the generated keys. A full external-CA pipeline—offline signing, distribution, and rotation policy—is out of scope here.

What's Next

• Upgrade a Kubernetes Cluster with kubeadm
• Kubernetes RBAC with Examples
• Kubernetes Authentication, Authorization and Admission Control

References

• Certificate Management with kubeadm — renew, external CA, and CSR generation
• kubeadm certs reference — check-expiration, renew, generate-csr
• Issue a Certificate for a Kubernetes API Client Using a CertificateSigningRequest — direct support for the Alice workflow
• Certificates and Certificate Signing Requests — CSR permissions, approval, signing, and asynchronous issuance
• Manage TLS Certificates in a Cluster — CertificateSigningRequest API
• kubectl wait — waiting for a non-empty JSONPath value
• kubeadm upgrade — automatic certificate renewal during control-plane upgrades
• Manual Rotation of CA Certificates — required continuation because kubeadm certs renew does not rotate CAs
• PKI certificates and requirements — required certificates for Kubernetes components

Summary

You mapped /etc/kubernetes/pki to the connections each certificate protects, read identities out of kubeconfig files, and used kubeadm certs check-expiration plus OpenSSL to see subjects and dates. Creating user alice showed the full client path: OpenSSL CSR, Kubernetes CertificateSigningRequest, approve, extract, and kubeconfig—then a Forbidden response until RBAC catches up.

kubeadm certs renew all rewrote every managed leaf and embedded kubeconfig certificate, but the cluster only used them after static Pods restarted. Renewing only admin.conf updated the control-plane kubeconfig without touching component certificates—refreshing $HOME/.kube/config from the renewed file closed the gap between the control-plane file and the copy you actually type against. When trust errors appear, check the clock, the CA blob, SANs, and whether the process still holds an old certificate in memory before you regenerate anything else.