Error using dev services with non-root containers

Help
1

I have a service (xiam-fpm-server) that is dependent on two additional services (xiam-web-server and xiam-frontend). I have remote development working when configuring each of the three services as separate entries in the manifest, and bringing each one up with a separate okteto up command in a separate terminal.

I’m now attempting to make use of the services key in the manifest to bring all three services up with a single okteto up invocation. However, while the primary dev container was coming up successfully, the xiam-web-server dev container would fail to come up, with the following error:

can't create '/var/okteto/remote/authorized_keys': File exists

I suspected that this was due to the fact that all three containers are configured to use a non-root user, and each one was using a different userid. I updated all three containers to use the same uid (777) and gid (777), to see if that helped. It did change the error message. Now it’s the primary dev container (xiam-fpm-server) that fails to come up, with the following error:

USER:777
2024-04-28T04:49:02 development container starting
2024-04-28T04:49:02 Copying secret authorized_keys to /var/okteto/remote/authorized_keys
cp: can't create '/var/okteto/remote/authorized_keys': Permission denied

Here are the contents of my manifest, in both configurations:

Separate services, working correctly:

build:
  fpm-server:
    target: fpm_server_dev

  web-server:
    target: web_server

  frontend:
    target: frontend

  cron:
    target: cron

  cli:
    target: cli_dev


deploy:
  remote: true
  commands:
    - helm dependency update "helm"
    - >-
      helm upgrade --install "xiam" "helm"
      --set okteto.enabled=true
      --set okteto.private=true
      --set environment="development"
      --set instance="${OKTETO_NAMESPACE}"
      --set productionMode=false
      --set configSecret.enabled=true
      --set configSecret.create=true
      --set cronjobResource.enabled=false
      --set cronjobDeployment.enabled=true
      --set ingress.enabled=true
      --set ingressRoute.internal.enabled=false
      --set ingressRoute.external.enabled=false
      --set imageRegistry="${OKTETO_BUILD_WEB_SERVER_REGISTRY}"
      --set images.webServer.repository="${OKTETO_BUILD_WEB_SERVER_REPOSITORY}"
      --set images.webServer.tag="${OKTETO_BUILD_WEB_SERVER_TAG}"
      --set images.fpmServer.repository="${OKTETO_BUILD_FPM_SERVER_REPOSITORY}"
      --set images.fpmServer.tag="${OKTETO_BUILD_FPM_SERVER_TAG}"
      --set images.frontend.repository="${OKTETO_BUILD_FRONTEND_REPOSITORY}"
      --set images.frontend.tag="${OKTETO_BUILD_FRONTEND_TAG}"
      --set images.cron.repository="${OKTETO_BUILD_CRON_REPOSITORY}"
      --set images.cron.tag="${OKTETO_BUILD_CRON_TAG}"
      --set images.cli.repository="${OKTETO_BUILD_CLI_REPOSITORY}"
      --set images.cli.tag="${OKTETO_BUILD_CLI_TAG}"
      --set frontend.enabled=true
      --set hostname="xiam-${OKTETO_NAMESPACE}.dev.xapiens.net"
      --set config.app.url="https://xiam-${OKTETO_NAMESPACE}.dev.xapiens.net"
      --set config.app.timezone="America/New_York"
      --set config.app.forceHttps=true
      --set database.create=true
      --set database.enabled=true
      --set config.db.hostname="${XIAM_DB_HOSTNAME:-xiam-database}"
      --set config.db.username="${XIAM_DB_USERNAME:-$OKTETO_NAMESPACE}"
      --set config.db.password="${XIAM_DB_PASSWORD}"
      --set config.db.name="${XIAM_DB_NAME:-$OKTETO_NAMESPACE}"
      --set redis.enabled=true
      --set redis.create=true
      --set config.redis.host=xiam-redis

dev:
  xiam-fpm-server:
    command: [ "php-fpm" ]
    reverse:
      - 9003:9003
    sync:
      - ".:/app"

    securityContext:
      runAsUser: 777
      runAsGroup: 777
      fsGroup: 777
  
  xiam-frontend:
    command: [ "npm", "run", "dev" ]
    forward:
      - 5173:5173
    sync:
      - ".:/app"

    securityContext:
      runAsUser: 777
      runAsGroup: 777
      fsGroup: 777

  xiam-web-server:
    command: [ "/docker-entrypoint.sh", "nginx", "-g", "daemon off;" ]
    sync:
      - "public:/app/public"

    securityContext:
      runAsUser: 777
      runAsGroup: 777
      fsGroup: 777

Here’s the dev section, reworked to use the services key (not working) :

dev:
  xiam-fpm-server:
    command: [ "php-fpm" ]
    forward:
      - 5173:xiam-frontend:5173
    reverse:
      - 9003:9003
    sync:
      - ".:/app"

    securityContext:
      runAsUser: 777
      runAsGroup: 777
      fsGroup: 777

    services:
      - name: xiam-web-server
        sync:
          - "public:/app/public"

      - name: xiam-frontend
        sync:
          - ".:/app"

Any input is appreciated. Thanks!

2

Hi @dmccorma,

After reviewing your manifest configuration, I suspect that the issue you’re experiencing might be related to the security context being applied after starting an okteto up session.

To resolve this issue, please try the following:

  1. Run okteto down -v to delete the dev containers.
  2. Then, run okteto up again to recreate the containers using the updated manifest (with the security context applied).

By deleting and recreating the containers, you’ll ensure that all files are created with the correct permissions from the start.

3

Hi @provecho . Thanks for the response. I ran okteto down -v, and confirmed via the okteto web UI that the dev volume was removed. I then ran okteto up, but received the same error in the dev container for the xiam-fpm-server service.

USER:777
2024-04-29T14:02:05 development container starting
2024-04-29T14:02:05 Copying secret authorized_keys to /var/okteto/remote/authorized_keys
cp: can't create '/var/okteto/remote/authorized_keys': Permission denied

Additionally, I’m not sure if it’s relevant, but while the xiam-web-server dev container seemed to come up correctly, the xiam-frontend container is also failing:

npm ERR! code ENOENT
npm ERR! syscall open
npm ERR! path /app/package.json
npm ERR! errno -2
npm ERR! enoent Could not read package.json: Error: ENOENT: no such file or directory, open '/app/package.json'
npm ERR! enoent This is related to npm not being able to find a file.
npm ERR! enoent 

npm ERR! A complete log of this run can be found in: /root/.npm/_logs/2024-04-29T14_02_46_571Z-debug-0.log

This suggests that for some reason my application’s files are not present in the container, though that doesn’t make much sense to me, since it should be using the same image that the regular deployment uses, which works properly. Anyway, since the main container isn’t coming up properly, it wouldn’t surprise me if any errors with the other dependent service containers are related to that, so this may be a red herring. But I just thought I’d share the observation in case it helps with troubleshooting.

Any other ideas? Thanks.

4

Hi @dmccorma,

Could you provide a real example by setting up an open-source repository where this issue occurs? Having access to a reproducible case will enable us to investigate further and provide a solution. Thank you for your cooperation!