Volume issue when trying to make use of dev services

Help
1

Hello. My application is written in PHP, and uses two separate deployments – a www-web deployment, running an nginx container to serve static files, and a www-php deployment, running php-fpm for serving dynamic content. In my existing Okteto manifest, I have the www-web and www-php services defined as two separate entries in the dev section:

# ...
dev:
  www-web:
    command: [ "/docker-entrypoint.sh", "nginx", "-g", "daemon off;" ]
    sync:
      - "src/extranet/web/src/wwwroot:/app/wwwroot"
      - "src/php-web-library:/app/php-web-library"

  www-php:
    command: [ "php-fpm" ]
    reverse:
      - 9003:9003
    sync:
      - "src/extranet/web/src/wwwroot:/app/wwwroot"
      - "src/php-web-library:/app/php-web-library"
    volumes:
      - /root/.composer/cache#
# ...

When I want to debug the application, I open two terminals and run okteto up www-web and okteto up www-php in parallel. This works fine.

While browsing the documentation for the Okteto manifest, I just came across the services sub-entry, which allows running additional services alongside the one being developed. This seems perfect for my use case, since I need to sync my local files to both pods, but I only need to establish tunnels to the www-php pod.

I’ve tested rewriting my manifest to make use of this feature:

# ...
dev:
  www-php:
    command: [ "php-fpm" ]
    reverse:
      - 9003:9003
    sync:
      - "src/extranet/web/src/wwwroot:/app/wwwroot"
      - "src/php-web-library:/app/php-web-library"
    volumes:
      - /root/.composer/cache#

    services:
      - name: www-web
        sync:
          - "src/extranet/web/src/wwwroot:/app/wwwroot"
          - "src/php-web-library:/app/php-web-library"
# ...

When I run okteto up www-php, it seems to succeed. However, the www-web-okteto pod remains in the pending state and is never scheduled. When I run kubectl describe pod on it, the “Events” section contains the following:

Events:
  Type     Reason            Age                    From               Message
  ----     ------            ----                   ----               -------
  Warning  FailedScheduling  5m47s (x3 over 5m51s)  default-scheduler  0/6 nodes are available: 1 Too many pods, 1 node(s) didn't match pod affinity rules, 4 node(s) didn't find available persistent volumes to bind. preemption: 0/6 nodes are available: 1 No preemption victims found for incoming pod, 5 Preemption is not helpful for scheduling..
  Warning  FailedScheduling  11s (x5 over 5m42s)    default-scheduler  0/6 nodes are available: 1 Too many pods, 2 node(s) didn't match pod affinity rules, 3 node(s) had volume node affinity conflict. preemption: 0/6 nodes are available: 1 No preemption victims found for incoming pod, 5 Preemption is not helpful for scheduling..

Examining the PVC for the development volume, I notice that the Access Mode is set to ReadWriteOnce, but it’s bound to both pods:

Name:          www-okteto
Namespace:     dmccorma
StorageClass:  gp3
Status:        Bound
Volume:        pvc-4dc2a66b-b73a-4b57-a1b1-c962e02f191b
Labels:        dev.okteto.com=true
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: ebs.csi.aws.com
               volume.kubernetes.io/selected-node: ip-10-60-2-184.ec2.internal
               volume.kubernetes.io/storage-provisioner: ebs.csi.aws.com
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      5Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Used By:       www-php-okteto-59466d4f6-l4tf6
               www-web-okteto-57bb757b5b-xffx9

My theory is that the second pod is unable to bind the PVC, and so it’s unable to be scheduled. Is the expectation that the development volume should be configured with a StorageClass that supports ReadWriteMany? Or am I barking up the wrong tree? Thanks.

2

I’ve made some progress, and believe I may have identified a bug in Okteto.

I realized that my theory about the issue being related to the ReadWriteOnce access mode on the PVC was incorrect. I had forgotten that ReadWriteOnce restricts a volume to be mounted by only a single node, not by a single pod. Sure enough, I did test reconfiguring my Okteto manifest to use an EFS volume for the persistentVolume, but it did not resolve the issue.

I see that Okteto is making use of pod affinities to ensure the development nodes get scheduled on the same node as each other, so that they can all mount the PVC. However, here’s where I think the problem was. I believe it was generating incorrect pod affinities when the dev section of the Okteto manifest is defined using selectors rather than deployment names.

My deployment names are somewhat long (xiam-fpm-server and xiam-web-server), and I didn’t want to type the whole name when running okteto up, so I was making use of selectors to let me have a shorter name. For reference, here is my entire manifest as it looked when the problem was occurring:

build:
  fpm-server:
    target: fpm_server
    args:
      DEV_ENVIRONMENT: true

  fpm-server-dev:
    target: fpm_server_dev
    args:
      DEV_ENVIRONMENT: true

  web-server:
    target: web_server
    args:
      DEV_ENVIRONMENT: true

  frontend:
    target: frontend
    args:
      DEV_ENVIRONMENT: true

  cron:
    target: cron
    args:
      DEV_ENVIRONMENT: true

  cli:
    target: cli
    args:
      DEV_ENVIRONMENT: true



deploy:
  - helm dependency update "helm"
  - >-
    helm upgrade --install "xiam" "helm"
    --set okteto.enabled=true
    --set okteto.private=true
    --set environment="development"
    --set instance="${OKTETO_NAMESPACE}"
    --set productionMode=false
    --set configSecret.enabled=true
    --set configSecret.create=true
    --set cronjobResource.enabled=false
    --set cronjobDeployment.enabled=true
    --set ingress.enabled=true
    --set ingressRoute.internal.enabled=false
    --set ingressRoute.external.enabled=false
    --set imageRegistry="${OKTETO_BUILD_WEB_SERVER_REGISTRY}"
    --set images.webServer.repository="${OKTETO_BUILD_WEB_SERVER_REPOSITORY}"
    --set images.webServer.tag="${OKTETO_BUILD_WEB_SERVER_TAG}"
    --set images.fpmServer.repository="${OKTETO_BUILD_FPM_SERVER_DEV_REPOSITORY}"
    --set images.fpmServer.tag="${OKTETO_BUILD_FPM_SERVER_DEV_TAG}"
    --set images.frontend.repository="${OKTETO_BUILD_FRONTEND_REPOSITORY}"
    --set images.frontend.tag="${OKTETO_BUILD_FRONTEND_TAG}"
    --set images.cron.repository="${OKTETO_BUILD_CRON_REPOSITORY}"
    --set images.cron.tag="${OKTETO_BUILD_CRON_TAG}"
    --set images.cli.repository="${OKTETO_BUILD_CLI_REPOSITORY}"
    --set images.cli.tag="${OKTETO_BUILD_CLI_TAG}"
    --set frontend.enabled=true
    --set hostname="xiam-${OKTETO_NAMESPACE}.dev.xapiens.net"
    --set config.app.url="https://xiam-${OKTETO_NAMESPACE}.dev.xapiens.net"
    --set config.app.timezone="America/New_York"
    --set config.app.forceHttps=true
    --set database.create=true
    --set database.enabled=true
    --set config.db.hostname="${XIAM_DB_HOSTNAME:-xiam-database}"
    --set config.db.username="${XIAM_DB_USERNAME:-$OKTETO_NAMESPACE}"
    --set config.db.password="${XIAM_DB_PASSWORD}"
    --set config.db.name="${XIAM_DB_NAME:-$OKTETO_NAMESPACE}"
    --set redis.enabled=true
    --set redis.create=true
    --set config.redis.host=xiam-redis


dev:
  fpm:
    selector:
      app.kubernetes.io/name: "xiam"
      app.kubernetes.io/instance: "${OKTETO_NAMESPACE}"
      app.kubernetes.io/component: "fpm-server"

    command: [ "sh" ]
    reverse:
      - 9003:9003
    sync:
      - ".:/app"
    volumes:
      - /root/.composer/cache#
      - /app/storage

    services:
      - selector:
          app.kubernetes.io/name: "xiam"
          app.kubernetes.io/instance: "${OKTETO_NAMESPACE}"
          app.kubernetes.io/component: "web-server"
        sync:
          - "public:/app/public"

And, just to recap, When I ran okteto up, I was seeing the development pod for the fpm-server deployment come up successfully, and the development pod for the web-server service was created but stayed in the “Pending” state and never got scheduled. The “Events” section of the latter pod’s kubectl desscribe output said this:

$ kubectl describe pod xiam-web-server-okteto-5458b7b64b-8vmn7

# ...

Events:
  Type     Reason            Age                      From               Message
  ----     ------            ----                     ----               -------
  Warning  FailedScheduling  3m23s (x236 over 7h21m)  default-scheduler  0/6 nodes are available: 6 node(s) didn't match pod affinity rules. preemption: 0/6 nodes are available: 6 Preemption is not helpful for scheduling.

The message indicated that the pod affinity is what was preventing the pod from being scheduled. Here are the affinities that were set on the pod:

nodeAffinity:
  preferredDuringSchedulingIgnoredDuringExecution:
    - weight: 50
      preference:
        matchExpressions:
          - key: dev.okteto.com/overloaded
            operator: DoesNotExist
    - weight: 100
      preference:
        matchExpressions:
          - key: dev.okteto.com/many-volumes
            operator: DoesNotExist
podAffinity:
  requiredDuringSchedulingIgnoredDuringExecution:
    - labelSelector:
        matchLabels:
          interactive.dev.okteto.com: fpm
      topologyKey: kubernetes.io/hostname
    - labelSelector:
        matchLabels:
          app.kubernetes.io/component: fpm-server
          app.kubernetes.io/instance: xiam
          app.kubernetes.io/managed-by: Helm
          app.kubernetes.io/name: xiam
          app.kubernetes.io/part-of: xiam
          app.kubernetes.io/version: 2.0.0
          app.xapiens.net/environment: development
          dev.okteto.com/affinity: xiam
          dev.okteto.com/deployed-by: xiam
          dev.okteto.com/user: <REDACTED>
          helm.sh/chart: xiam-2.0.0
          interactive.dev.okteto.com: fpm-server
      topologyKey: kubernetes.io/hostname
  preferredDuringSchedulingIgnoredDuringExecution:
    - weight: 10
      podAffinityTerm:
        labelSelector:
          matchExpressions:
            - key: dev.okteto.com/affinity
              operator: In
              values:
                - xiam
        topologyKey: kubernetes.io/hostname

As you can see, it had conflicting label selectors for the interactive.dev.okteto.com label. The first one was looking for the name of the key in my manifest (fpm), while the second was looking for fpm-server). The actual value on the development pod for fpm-server was fpm-server.

When I cleared everything out and started fresh (okteto down, okteto destroy, and destroy the development volume in the web UI), and reworked my manifest to use the full deployment names rather than selectors, it seemed to work correctly. Here is the working version of the dev section of my manifest:

# ...

dev:
  xiam-fpm-server:
    command: [ "sh" ]
    reverse:
      - 9003:9003
    sync:
      - ".:/app"
    volumes:
      - /root/.composer/cache#
      - /app/storage

    services:
      - name: xiam-web-server
        sync:
          - "public:/app/public"

However, I would still like to get it working using selectors, so I can use a shorter name. Can someone confirm whether this is a bug, or working as intended? Thanks.

3

Hey @dmccorma, thanks for the through explanation of your issue.

Let me try and take it one step at a time:

I realized that my theory about the issue being related to the ReadWriteOnce access mode on the PVC was incorrect. I had forgotten that ReadWriteOnce restricts a volume to be mounted by only a single node , not by a single pod

Your observation is correct. When running ‘services’, Okteto uses affinities and tolerations to ensure that both pods land on the same node. That way, both dev containers can share the same volume. That way, okteto only needs to synchronize the changes once, and all dev containers have the same files.

1 Like
4

Yes, this is looks like a bug. The services feature should work the same regardless of whether you have a selector or the service name. I’ll follow up with the team internally to get more information on this and post the result back to this thread.

1 Like