Node affinity blocking scheduling

Okteto Self-Hosted
1

Hey y’all, sorry to keep pestering. We’re seeing an odd behavior with just one of our okteto manifests - the dev image/container can’t start, due to a Kubernetes scheduling issue. We’re getting this error:

0/2 nodes are available: 2 node(s) didn't match Pod's node affinity/selector. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling.

We don’t have any node selectors. There is this:

Labels: dev.okteto.com/affinity=bennidhamma

Could that be used as a node selector label? We’ve been poking at it for a few days now. Any ideas?

2

Hi @benjoldersma ,

That label shouldn’t prevent the pod to be scheduled as that is used as preferred affinity, not a required one. So if there is no a node available that matches that, it can go to other node without problems. There has to be some other thing. Could you share the affected pod spec removing environment variables and any other sensible information? I’m specially interested on checking the spec.affinity section and nodeSelector.

Thanks in advance!

3

OK, and please let me know if we’ve veered into general kubernetes territory on this. I don’t want to lean on y’all for that!

but I do think this might be related to okteto.

I see this:

      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: topology.kubernetes.io/zone
            operator: In
            values:
            - us-west-2a

Which, I had seen before, but just new reviewing, we only have two nodes (we’re trying to keep costs down!!) one in 2b, one in 2c, but we don’t have one in 2a.

The question is: how is this rule getting into the pod spec? I can’t find it anywhere in our repo…

4

We don’t usually add required node affinities to any pods to prevent scheduling issues. The only case in which we would add something is if this configuration setting is specified in the chart values. Could you check if at some point you added something there?

If that’s not the case, then, I would say it is some Kubernetes controller adding that for some reason. Did you have any node in the zone a at some point? Is it a pod of any Okteto component or is it a pod of some of your dev environments?

While you confirm in you have set the setting mentioned above, I’ll try to see possible reasons why that affinity could have been added

5

I don’t have anything about affinity, or zone, or us-west-2a in any config file I can find :frowning:

We did have a node there. We’re using spot instances, so they get moved around a bit. it’s not an okteto component, but rather the okteto dev container that overrides the non-dev container…

(the one that comes from this part of okteto.yml:

# The dev section defines how to activate a development container
# More info: https://www.okteto.com/docs/reference/manifest/#dev
dev:
  loriini-web:
    command: sh
    workdir: /app
    sync:
      - .:/app
    environment:
      - name=$USER
      - NODE_ENV=development
    forward:
      - 8080:80

I dont think we have any other controllers. it’s basically just a default EKS install with okteto on it…

6

Ok, then, there might a case in which is automatically done by EKS. Could that pod had been deployed in the zone a originally? If that is the case, it might happen that the persistent volume in the provider side was in the a zone.

As the volume is in that zone, any pod with that volume attached has to be deployed in the same zone where the volume is, but as there are no nodes deployed there, the pod is stuck because cannot be scheduled.

One thing to verify this is checking the PVC attached to that pod. So you can execute okteto get pvc -n <namespace> to identify which is the PVC for the affected pod (as it is the PVC for a development pod it has to have the suffix okteto, if not, you can see the pod to check which PVC is attached).

Once you have the PVC, you can get the PersistentVolume and see its manifest kubectl get pv <volume-name> -n <namespace> -o yaml and you should get a label indicating the region or zone if I’m not wrong. You can see then if it’s in zone a.

If that is the case, you would have to execute okteto down -v, so the persistent volume is deleted and execute again okteto up, so a new volume is created in one of the zones where you have a node and the pods should be able to schedule.

Try this and let me know if this works

7

phew. got it! i think i must have updated the storage class at some point. it had a node affinity for 2a, 2b and 2c - must have been when we had 3 nodes, one in each. removed node 2a and we’re back in business.

thanks for the help Nacho! Nothing to do with okteto. just my own clumsy k8s-fu.

1 Like
8

:clap: Glad to hear it is solved now! I always struggle with affinities and anti affinities. I still feel I don’t understand them 100% :sweat:

1 Like