Okteto pods are freezing for us after upgrading to EKS v1.23

okteto pods are freezing for us after upgrading to EKS v1.23, more details in the thread. would appreciate your help

okteto version: 1.13.11
kubernetes version: 1.23
EKS version 1.23

  • pods are getting stuck on okteto down -v
  • pods are getting stuck when running npm run start:dev

will attach events and okteto.yaml


events:
11m         Warning   FailedMount              pod/webapp-d54c45b56-qt8l2            MountVolume.SetUp failed for volume "config" : failed to sync configmap cache: timed out waiting for the condition
11m         Normal    Pulling                  pod/webapp-d54c45b56-qt8l2            Pulling image "723128751635.dkr.ecr.us-east-1.amazonaws.com/webapp:latest"
11m         Normal    Pulled                   pod/webapp-d54c45b56-qt8l2            Successfully pulled image "723128751635.dkr.ecr.us-east-1.amazonaws.com/webapp:latest" in 609.607842ms
11m         Normal    Created                  pod/webapp-d54c45b56-qt8l2            Created container app
11m         Normal    Started                  pod/webapp-d54c45b56-qt8l2            Started container app
11m         Normal    SuccessfulDelete         replicaset/webapp-5f6cd7b77           Deleted pod: webapp-5f6cd7b77-nsxmf
11m         Normal    ScalingReplicaSet        deployment/webapp                     Scaled down replica set webapp-5f6cd7b77 to 0
8m9s        Normal    Killing                  pod/webapp-5f6cd7b77-nsxmf            Stopping container app
8m7s        Normal    TaintManagerEviction     pod/webapp-5f6cd7b77-nsxmf            Cancelling deletion of Pod camel/webapp-5f6cd7b77-nsxmf
8m          Normal    WaitForFirstConsumer     persistentvolumeclaim/okteto-webapp   waiting for first consumer to be created before binding
8m2s        Normal    Killing                  pod/webapp-d54c45b56-qt8l2            Stopping container app
8m2s        Normal    ScalingReplicaSet        deployment/webapp                     Scaled down replica set webapp-d54c45b56 to 0
8m2s        Normal    SuccessfulDelete         replicaset/webapp-d54c45b56           Deleted pod: webapp-d54c45b56-qt8l2
8m          Normal    ScalingReplicaSet        deployment/webapp                     Scaled up replica set webapp-566458858 to 1
8m          Normal    ExternalProvisioning     persistentvolumeclaim/okteto-webapp   waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
8m          Normal    Provisioning             persistentvolumeclaim/okteto-webapp   External provisioner is provisioning volume for claim "camel/okteto-webapp"
8m          Normal    SuccessfulCreate         replicaset/webapp-566458858           Created pod: webapp-566458858-42tnv
7m57s       Normal    ProvisioningSucceeded    persistentvolumeclaim/okteto-webapp   Successfully provisioned volume pvc-535e55a7-b955-4e85-802e-e1b0e8cff51b
7m56s       Normal    Scheduled                pod/webapp-566458858-42tnv            Successfully assigned camel/webapp-566458858-42tnv to ip-192-168-53-194.ec2.internal
7m54s       Normal    SuccessfulAttachVolume   pod/webapp-566458858-42tnv            AttachVolume.Attach succeeded for volume "pvc-535e55a7-b955-4e85-802e-e1b0e8cff51b"
7m50s       Normal    Created                  pod/webapp-566458858-42tnv            Created container okteto-bin
7m50s       Normal    Started                  pod/webapp-566458858-42tnv            Started container okteto-bin
7m50s       Normal    Pulled                   pod/webapp-566458858-42tnv            Container image "okteto/bin:1.3.3" already present on machine
7m49s       Normal    Pulled                   pod/webapp-566458858-42tnv            Container image "okteto/node:14" already present on machine
7m49s       Normal    Created                  pod/webapp-566458858-42tnv            Created container okteto-init-volume
7m49s       Normal    Started                  pod/webapp-566458858-42tnv            Started container okteto-init-volume
7m48s       Normal    Pulling                  pod/webapp-566458858-42tnv            Pulling image "okteto/node:14"
7m48s       Normal    Pulled                   pod/webapp-566458858-42tnv            Successfully pulled image "okteto/node:14" in 160.08077ms
7m48s       Normal    Started                  pod/webapp-566458858-42tnv            Started container app
7m48s       Normal    Created                  pod/webapp-566458858-42tnv            Created container app

okteto.yaml:

name: webapp
image: okteto/node:14
command: bash
workdir: /app
sync:
  folders:
    - .:/app
  verbose: false
persistentVolume:
  size: 512Mi
secrets:
  - $HOME/.npmrc:/root/.npmrc:400
environment:
  - NODE_ENV=development
context: dev
services:
  - name: webapp
    sync:
      - dist:/usr/share/nginx/html

kubectl top:

NAME                     CPU(cores)   MEMORY(bytes)   
webapp-566458858-42tnv   16m          1241Mi

Can you share the output of this command? If you can run it with verbose logs, even better. Same for okteto up , if that one is also getting stuck.

Hi @evoosa,
What do you mean by pods getting stuck?
Last events indicate the the pod was started:

7m48s       Normal    Pulling                  pod/webapp-566458858-42tnv            Pulling image "okteto/node:14"
7m48s       Normal    Pulled                   pod/webapp-566458858-42tnv            Successfully pulled image "okteto/node:14" in 160.08077ms
7m48s       Normal    Started                  pod/webapp-566458858-42tnv            Started container app
7m48s       Normal    Created                  pod/webapp-566458858-42tnv 

also, are you using Okteto Self-hosted? or only the Okteto CLI?