Kubernetes Pod Warning: 1 Node(S) Had Volume Node Affinity Conflict

The error "volume node affinity conflict" happens when the persistent volume claims that the pod is using are scheduled on different zones, rather than on one zone, and so the actual pod was not able to be scheduled because it cannot connect to the volume from another zone. To check this, you can see the details of all the Persistent Volumes. To check that, first get your PVCs:

$ kubectl get pvc -n <namespace>

Then get the details of the Persistent Volumes (not Volume claims)

$  kubectl get pv

Find the PVs, that correspond to your PVCs and describe them

$  kubectl describe pv <pv1> <pv2>

You can check the Source.VolumeID for each of the PV, most likely they will be different availability zone, and so your pod gives the affinity error. To fix this, create a storageclass for a single zone and use that storageclass in your PVC.

kind: StorageClass
apiVersion: 
metadata:
  name: region1storageclass
provisioner: 
parameters:
  type: gp2
  encrypted: "true" # if encryption required
volumeBindingMode: WaitForFirstConsumer
allowedTopologies:
- matchLabelExpressions:
  - key: 
    values:
    - eu-west-2b # this is the availability zone, will depend on your cloud provider
    # multi-az can be added, but that defeats the purpose in our scenario

There a few things that can cause this error:

1. Node isn’t labeled properly. I had this issue on AWS when my worker node didn’t have appropriate labels(master had them though) like that:

   After patching the node with the labels, the “1 node(s) had volume node affinity conflict” error was gone, so PV, PVC with a pod were deployed successfully. The value of these labels is cloud provider specific. Basically, it is the job of the cloud provider(with —cloud-provider option defined in cube-controller, API-server, kubelet) to set those labels. If appropriate labels aren’t set, then check that your CloudProvider integration is correct. I used kubeadm, so it is cumbersome to set up but with other tools, kops, for instance, it is working right away.

2. Based on your PV definition and the usage of nodeAffinity field, you are trying to use a local volume, (read here local volume description link, official docs), then make sure that you set "NodeAffinity field" like that(it worked in my case on AWS):

   nodeAffinity:

        required:
         nodeSelectorTerms:
          - matchExpressions:
            - key: 
              operator: In
              values:
              - my-node  # it must be the name of your node(kubectl get nodes)
   

So that after creating the resource and running describe on it it will show up there like that:

         Required Terms:  
                    Term 0:   in [your node name]

3. StorageClass definition(named local-storage, which is not posted here) must be created with volumeBindingMode set to WaitForFirstConsumer for local storage to work properly. Refer to the example here storage class local description, official doc to understand the reason behind that.

0. If you didn't find the solution in other answers...

In our case the error happened on a AWS EKS cluster freshly provisioned with Pulumi (see full source here). The error drove me nuts, since I didn't change anything, just created a PersistentVolumeClaim as described in the Buildpacks Tekton docs:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: buildpacks-source-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 500Mi

I didn't change anything else from the default EKS configuration and also didn't add/change any PersistentVolume or StorageClass (in fact I didn't even know how to do that). As the default EKS setup seems to rely on 2 nodes, I got the error:

0/2 nodes are available: 2 node(s) had volume node affinity conflict.

Reading through Sownak Roy's answer I got a first glue what to do - but didn't know how to do it. So for the folks interested here are all my steps to resolve the error: