Fix Pending PVC, FailedMount and Volume Attachment Errors

When storage fails, the useful question is not whether storage is broken. It is which stage failed. This guide follows the PVC lifecycle in order and uses Events to separate intentional WaitForFirstConsumer delays from real provisioning, attach, and mount problems.

Identify the failure stage

Volume setup runs as a chain. Name the stage before changing YAML:

Immediate binding:

PVC → provision or match PV → bind → schedule Pod

WaitForFirstConsumer:

PVC + consumer Pod → scheduler selects a suitable node
                   → provision or match PV for that topology
                   → bind claim
                   → assign Pod

Both paths:

attach when required → mount → container starts

Map what you see to that chain:

Create a demo namespace for the examples below:

kubectl create ns pvc-errors-demo

namespace/pvc-errors-demo created

Inspect PVC, PV and StorageClass

Start with objects and Events, not guesses from the word Pending.

PVCs are namespaced. PVs are cluster-scoped, so list them separately:

kubectl get pvc -n pvc-errors-demo

On a fresh namespace this returns no PVCs until you create claims.

kubectl get pv

The cluster may already have PVs even when the demo namespace is empty. List StorageClasses next:

kubectl get storageclass

NAME         PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
local-path   rancher.io/local-path   Delete          WaitForFirstConsumer   false                  <age>

This lab uses local-path with WaitForFirstConsumer. Describe the class when a claim names it:

kubectl describe storageclass local-path

Name:            local-path
IsDefaultClass:  No
Provisioner:           rancher.io/local-path
VolumeBindingMode:     WaitForFirstConsumer

Sort Events in the namespace so the newest explanations rise last:

kubectl -n pvc-errors-demo get events --sort-by=.metadata.creationTimestamp

Treat Event messages as the primary diagnosis. Pending alone does not tell you whether the controller is waiting on purpose or failing to provision.

Sample output in this guide is based on the lab. Generated PV names, Pod names, node names, IP addresses, Event counts, ordering, and ages will differ between clusters.

Distinguish WaitForFirstConsumer from failure

Create a PVC that uses local-path and do not create a consumer Pod yet:

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: wffc-claim
  namespace: pvc-errors-demo
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: local-path
  resources:
    requests:
      storage: 1Gi
EOF

persistentvolumeclaim/wffc-claim created

kubectl -n pvc-errors-demo get pvc wffc-claim

NAME         STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
wffc-claim   Pending                                      local-path     <unset>                 <age>

Describe the claim and read Events:

kubectl -n pvc-errors-demo describe pvc wffc-claim

Events:
  Type    Reason                Age   From                         Message
  ----    ------                ----  ----                         -------
  Normal  WaitForFirstConsumer  <age> persistentvolume-controller  waiting for first consumer to be created before binding

That is intentional. Under WaitForFirstConsumer, provisioning waits for a consumer Pod so scheduling requirements can guide volume placement. The scheduler selects a candidate node, after which the provisioner creates a topology-compatible volume and binding completes.

Using spec.nodeName on the initial consumer bypasses this scheduler workflow, which is why Kubernetes specifically warns against it for unbound WaitForFirstConsumer claims. Create a compatible consumer and let the scheduler place it:

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: wffc-consumer
  namespace: pvc-errors-demo
spec:
  containers:
  - name: app
    image: busybox:1.36
    command: ["sh","-c","echo ok > /data/hello; sleep 3600"]
    volumeMounts:
    - name: data
      mountPath: /data
  volumes:
  - name: data
    persistentVolumeClaim:
      claimName: wffc-claim
EOF

pod/wffc-consumer created

kubectl -n pvc-errors-demo wait --for=condition=Ready pod/wffc-consumer --timeout=90s

pod/wffc-consumer condition met

kubectl -n pvc-errors-demo get pvc wffc-claim

NAME         STATUS   VOLUME      CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
wffc-claim   Bound    <pv-name>   1Gi        RWO            local-path     <unset>                 <age>

PVC Events advance from WaitForFirstConsumer to ProvisioningSucceeded. Full StorageClass binding-mode detail lives in StorageClass and dynamic provisioning.

Fix a missing or incorrect StorageClass

A different Pending claim with a bad class name is a real failure. Create one:

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: bad-sc-claim
  namespace: pvc-errors-demo
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: does-not-exist
  resources:
    requests:
      storage: 1Gi
EOF

persistentvolumeclaim/bad-sc-claim created

kubectl -n pvc-errors-demo describe pvc bad-sc-claim

Events:
  Type     Reason              Age              From                         Message
  ----     ------              ----             ----                         -------
  Warning  ProvisioningFailed  <age> (x2 over <age>)  persistentvolume-controller  storageclass.storage.k8s.io "does-not-exist" not found

A PVC specification is effectively immutable after creation, apart from limited fields such as requested storage and volumeAttributesClassName under applicable conditions. storageClassName cannot normally be corrected in place.

Delete and recreate this still-unbound demo claim with a valid storageClassName. Alternatively, omit storageClassName when the cluster has an appropriate default StorageClass. Explicitly setting storageClassName: "" requests no StorageClass and prevents default-class assignment. An omitted class and an explicitly empty class have different behavior.

kubectl delete pvc bad-sc-claim -n pvc-errors-demo

Recreate the claim with a real class:

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: bad-sc-claim
  namespace: pvc-errors-demo
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: local-path
  resources:
    requests:
      storage: 1Gi
EOF

Do not create a StorageClass that names a provisioner you never installed. That only moves the failure to ExternalProvisioning forever.

Also verify:

• Default StorageClass annotations when the claim omits storageClassName
• Provisioner Pod health (this lab: local-path-storage)
• StorageClass parameters the driver requires
• Spelling of the provisioner string

Fix static PV binding mismatches

Static claims bind only when capacity, access modes, volume mode, StorageClass, and selectors match. Create a 500Mi PV and a 1Gi claim with empty StorageClass. Add a unique label and matching selector so another unrelated PV cannot satisfy the claim:

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: PersistentVolume
metadata:
  name: static-small-pv
  labels:
    demo: static-size-mismatch
spec:
  capacity:
    storage: 500Mi
  accessModes: ["ReadWriteOnce"]
  persistentVolumeReclaimPolicy: Retain
  storageClassName: ""
  hostPath:
    path: /tmp/static-small-pv
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: static-mismatch
  namespace: pvc-errors-demo
spec:
  selector:
    matchLabels:
      demo: static-size-mismatch
  accessModes: ["ReadWriteOnce"]
  storageClassName: ""
  resources:
    requests:
      storage: 1Gi
EOF

persistentvolume/static-small-pv created
persistentvolumeclaim/static-mismatch created

kubectl -n pvc-errors-demo describe pvc static-mismatch

Events:
  Type    Reason         Age   From                         Message
  ----    ------         ----  ----                         -------
  Normal  FailedBinding  <age> persistentvolume-controller  no persistent volumes available for this claim and no storage class is set

The PV stays Available because 500Mi cannot satisfy a 1Gi request. The selector restricts this claim to the intentionally undersized PV, so another unrelated PV cannot satisfy it. PVC selectors and StorageClass requirements are evaluated together.

Compare these fields on every stuck static claim:

• Capacity
• Access modes
• Volume mode
• StorageClass (including empty string vs unset)
• Label selector
• Existing claimRef on a Released PV

Static provisioning concepts are covered in PersistentVolume and PVC.

Inspect CSI components

Start by mapping the claim to its PV and volume source. Do not inspect random CSI objects before you know which driver or provisioner owns the claim:

NS=pvc-errors-demo
PVC=wffc-claim

PV=$(kubectl get pvc -n "$NS" "$PVC" \
  -o jsonpath='{.spec.volumeName}')

kubectl get pv "$PV" \
  -o jsonpath='csiDriver={.spec.csi.driver}{"\n"}hostPath={.spec.hostPath.path}{"\n"}localPath={.spec.local.path}{"\n"}'

For this article's local-path claim, the CSI driver field is empty because the resulting PV uses a local filesystem volume source. Derive the driver and inspect CSI objects only when the PV is CSI-backed:

DRIVER=$(kubectl get pv "$PV" \
  -o jsonpath='{.spec.csi.driver}')

if [ -z "$DRIVER" ]; then
  echo "PV $PV is not CSI-backed"
else
  kubectl get csidriver "$DRIVER" \
    -o custom-columns=NAME:.metadata.name,ATTACH_REQUIRED:.spec.attachRequired

  kubectl get volumeattachment \
    -o custom-columns=NAME:.metadata.name,PV:.spec.source.persistentVolumeName,NODE:.spec.nodeName,ATTACHED:.status.attached,ATTACH_ERROR:.status.attachError.message
fi

Sample output for this lab:

PV <pv-name> is not CSI-backed

Kubernetes uses CSIDriver.spec.attachRequired to determine whether the attach operation and VolumeAttachment workflow apply. A VolumeAttachment is expected only when the CSI driver requires a controller attach operation. Drivers with spec.attachRequired: false skip this stage.

Confirm the provisioner that owns your StorageClass is Running:

kubectl get pods -n local-path-storage -o wide

NAME                                              READY   STATUS    RESTARTS   AGE   IP         NODE         NOMINATED NODE   READINESS GATES
local-path-provisioner-<replicaset-hash>-<pod-suffix>   1/1     Running   0          <age>  <pod-ip>   <node-name>  <none>           <none>

When diagnosing CSI drivers, also identify the controller Pods, node plugin on the selected node, external provisioner, external attacher, and driver registration DaemonSet.

Fix FailedAttachVolume and Multi-Attach

FailedAttachVolume means the attach path failed before a successful mount. Classic causes:

• A ReadWriteOnce volume that requires attachment can produce Multi-Attach when Kubernetes tries to use it from another node. ReadWriteOnce still permits multiple Pods on the same node.
• ReadWriteOncePod restricts the claim to one Pod across the cluster, and conflicts are normally prevented during scheduling with a FailedScheduling event. ReadWriteOncePod is CSI-only and has different enforcement from node-scoped ReadWriteOnce.
• Stale Pod stuck Terminating on the old node
• CSI controller or attacher down
• Node NotReady or cloud-provider attach timeout

On this lab, local-path volumes are node-local. Rancher local-path volumes carry hostname affinity and are accessible only from their provisioned node. If a second Pod is pinned to a different node than the Bound PV's topology, kubelet reports a mount-time topology failure instead of a cloud Multi-Attach string.

Derive the volume's current node and select another Ready node:

VOLUME_NODE=$(kubectl get pod wffc-consumer \
  -n pvc-errors-demo \
  -o jsonpath='{.spec.nodeName}')

WRONG_NODE=$(kubectl get nodes --no-headers |
  awk -v current="$VOLUME_NODE" \
  '$1 != current && $2 ~ /^Ready/ {print $1; exit}')

test -n "$WRONG_NODE" || {
  echo "A second Ready node is required for this test"
  exit 1
}

Pin a second Pod to the other node:

kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: wrong-node
  namespace: pvc-errors-demo
spec:
  nodeName: $WRONG_NODE
  containers:
  - name: app
    image: busybox:1.36
    command: ["sleep", "3600"]
    volumeMounts:
    - name: data
      mountPath: /data
  volumes:
  - name: data
    persistentVolumeClaim:
      claimName: wffc-claim
EOF

pod/wrong-node created

Wait for the kubelet to record a FailedMount event before you run describe:

kubectl events -n pvc-errors-demo \
  --for pod/wrong-node \
  --types=Warning \
  --watch

Stop with Ctrl+C after FailedMount appears, then run:

kubectl describe pod wrong-node -n pvc-errors-demo

Events:
  Type     Reason       Age               From     Message
  ----     ------       ----              ----     -------
  Warning  FailedMount  <age> (x5 over <age>)  kubelet  MountVolume.NodeAffinity check failed for volume "<pv-name>" : no matching NodeSelectorTerms

The PV carries hostname affinity to the node where the first consumer was scheduled. Delete the wrong Pod and keep the workload on the node that owns the volume:

kubectl delete pod wrong-node -n pvc-errors-demo

For CSI or cloud disks, Multi-Attach events look like "Volume is already exclusively attached to one node." Then:

1. Find Pods that still reference the claim.
2. Stop the previous consumer and wait for the CSI driver to detach the volume normally.
3. Confirm the backend no longer considers it attached.
4. Check kubectl get volumeattachment for the node that still holds the volume.
5. If the VolumeAttachment remains stale, follow the CSI driver or storage-provider recovery procedure. Manually deleting it should be a driver-specific corrective step, not the default Multi-Attach fix.

Deleting a VolumeAttachment initiates detach processing through the external attacher. It is not merely clearing harmless status. Forced detach while the previous node may still use the disk can risk filesystem or data corruption.

Fix FailedMount

FailedMount is kubelet saying MountVolume.SetUp failed. After PVC is Bound, check:

• CSI node plugin on the Pod’s node
• kubelet logs
• Node driver registration (CSINode)
• Secret or credential references for the driver
• Filesystem type and mount options
• Device presence and volume mode
• Node path for local volumes
• subPath values

Single-file subPath patterns are covered in subPath volume mounts.

Diagnose ConfigMap and Secret mount errors separately

Not every FailedMount is CSI. Missing projected volumes fail the same Event type. Create a Pod that mounts a ConfigMap that does not exist:

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: missing-cm
  namespace: pvc-errors-demo
spec:
  containers:
  - name: app
    image: busybox:1.36
    command: ["sleep","3600"]
    volumeMounts:
    - name: cfg
      mountPath: /cfg
  volumes:
  - name: cfg
    configMap:
      name: does-not-exist-cm
EOF

pod/missing-cm created

Wait for the kubelet to record a FailedMount event:

kubectl events -n pvc-errors-demo \
  --for pod/missing-cm \
  --types=Warning \
  --watch

Stop with Ctrl+C after FailedMount appears, then run:

kubectl describe pod missing-cm -n pvc-errors-demo

Events:
  Type     Reason       Age              From               Message
  ----     ------       ----             ----               -------
  Normal   Scheduled    <age>            default-scheduler  Successfully assigned pvc-errors-demo/missing-cm to <node-name>
  Warning  FailedMount  <age> (x5 over <age>)  kubelet            MountVolume.SetUp failed for volume "cfg" : configmap "does-not-exist-cm" not found

A missing Secret looks the same with a secret "…" not found message. A ConfigMap that exists but references a bad key fails like this:

kubectl -n pvc-errors-demo create configmap demo-cfg --from-literal=app.conf=hello

configmap/demo-cfg created

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: missing-key
  namespace: pvc-errors-demo
spec:
  containers:
  - name: app
    image: busybox:1.36
    command: ["sleep","3600"]
    volumeMounts:
    - name: cfg
      mountPath: /cfg
  volumes:
  - name: cfg
    configMap:
      name: demo-cfg
      items:
      - key: not-a-real-key
        path: app.conf
EOF

pod/missing-key created

Wait for the kubelet to record a FailedMount event:

kubectl events -n pvc-errors-demo \
  --for pod/missing-key \
  --types=Warning \
  --watch

Stop with Ctrl+C after FailedMount appears, then run:

kubectl describe pod missing-key -n pvc-errors-demo

Events:
  Type     Reason       Age              From               Message
  ----     ------       ----             ----               -------
  Normal   Scheduled    <age>            default-scheduler  Successfully assigned pvc-errors-demo/missing-key to <node-name>
  Warning  FailedMount  <age> (x5 over <age>)  kubelet            MountVolume.SetUp failed for volume "cfg" : configmap references non-existent config key: not-a-real-key

Create the object in the right namespace, fix the key name, or drop the bad items / subPath entry. Do not restart CSI controllers for these Events.

Use node-level logs

When Events name a node and kubelet mount step, read kubelet logs on that node:

journalctl -u kubelet --since "10 min ago" | grep -i -E 'Mount|CSI|volume' | tail -40

List CSI or provisioner containers on the node with crictl when you have SSH:

crictl ps -a |
  grep -Ei 'csi|provisioner|attacher|registrar|local-path'

That command helps locate the relevant node plugin, registrar, provisioner, or attacher container before you read its logs.

If SSH is unavailable but the node still schedules debug Pods, use kubectl debug node/<node> and inspect from the debug Pod’s host mount. Prefer Events first; drop to node logs when the Event text is generic or timed out.

Verify recovery

After a fix, walk the chain again:

1. PVC is Bound.
2. PV CLAIM points at the expected namespace/name.
3. VolumeAttachment shows attached when your driver requires controller attach (spec.attachRequired: true).
4. Pod schedules on a topology-compatible node.
5. FailedMount / FailedAttachVolume Events stop.
6. Container becomes Ready.
7. A read/write check on the mount path succeeds.

For the lab consumer:

kubectl -n pvc-errors-demo exec wffc-consumer -- cat /data/hello

ok

That confirms the volume mounted and the app can write.

Troubleshooting

Clean up

Delete the demo namespace and any leftover static PVs from this article:

kubectl delete ns pvc-errors-demo

kubectl delete pv static-small-pv rwo-shared-pv --ignore-not-found

Restore the control-plane NoSchedule taint if you removed it while testing node pins.

What's Next

• Kubernetes Networking Model, CNI and kube-proxy
• Kubernetes Services, Endpoints and EndpointSlices
• Troubleshoot a Kubernetes Service That Is Not Working

References

• Persistent Volumes
• Storage Classes
• VolumeAttachment API
• CSIDriver API
• Local Path Provisioner
• ConfigMaps
• Secrets
• Configure a Pod to Use a PersistentVolume for Storage

Summary

PVC problems become manageable when you name the stage: StorageClass selection, provision or bind, schedule, attach, then mount. Events tell you whether Pending is WaitForFirstConsumer waiting for a consumer Pod or ProvisioningFailed because the class does not exist.

A Bound claim is only halfway there. Local volumes can fail with NodeAffinity FailedMount when a Pod lands on the wrong node. CSI and cloud disks more often show FailedAttachVolume or Multi-Attach when ReadWriteOnce is still held elsewhere. ConfigMap and Secret mistakes also produce FailedMount. Read the Event text before blaming the CSI driver.

Fix the matching stage, then verify Bound status, attachment where applicable, clean Pod Events, and a simple read/write on the mount path. Next, tighten StorageClass binding mode choices or static PV field matching so the same failure does not return on the next claim.