externalVolumes with lots of files take forever to deploy

Steve · February 10, 2024, 2:35pm

Hi

I’m using the latest version to deploy. If I’ve got no externalVolumes or an empty one it deploys straight away. If I’ve got an externalVolume with hundreds of thousands of files, it goes away for hours before sometimes deploying. Any ideas?

dev:
  test:
    autocreate: true
    image:  busybox
    forward:
      - 8080:8080
      - 9229:9229
    workdir: /usr/src/app
    sync:
      - .:/usr/src/app
    externalVolumes:
      - historic::/external
    resources:
      requests:
        cpu: "500m"
        memory: "1000Mi"
      limits:
        cpu: "1000m"
        memory: "2000Mi"
    initContainer:
      image: okteto/bin:1.4.4
      resources:
        requests:
          cpu: "500m"
          memory: "1000Mi"
        limits:
          cpu: "1000m"
          memory: "2000Mi"

Steve · February 10, 2024, 3:23pm

The okteto.log file contains

time="2024-02-10T14:31:32Z" level=info msg="ticker event in 'waitUntilDevelopmentContainerIsRunning' loop..." action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T14:31:42Z" level=info msg="ticker event in 'waitUntilDevelopmentContainerIsRunning' loop..." action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T14:31:52Z" level=info msg="ticker event in 'waitUntilDevelopmentContainerIsRunning' loop..." action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T14:32:02Z" level=info msg="ticker event in 'waitUntilDevelopmentContainerIsRunning' loop..." action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T14:32:12Z" level=info msg="ticker event in 'waitUntilDevelopmentContainerIsRunning' loop..." action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T14:32:22Z" level=info msg="ticker event in 'waitUntilDevelopmentContainerIsRunning' loop..." action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
...
659fbb115bd6 version=2.25.1
time="2024-02-10T15:11:02Z" level=info msg="ticker event in 'waitUntilDevelopmentContainerIsRunning' loop..." action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:11:12Z" level=info msg="ticker event in 'waitUntilDevelopmentContainerIsRunning' loop..." action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:11:22Z" level=info msg="ticker event in 'waitUntilDevelopmentContainerIsRunning' loop..." action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:11:32Z" level=info msg="ticker event in 'waitUntilDevelopmentContainerIsRunning' loop..." action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:11:42Z" level=info msg="ticker event in 'waitUntilDevelopmentContainerIsRunning' loop..." action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:11:52Z" level=info msg="ticker event in 'waitUntilDevelopmentContainerIsRunning' loop..." action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:12:02Z" level=info msg="ticker event in 'waitUntilDevelopmentContainerIsRunning' loop..." action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:12:12Z" level=info msg="ticker event in 'waitUntilDevelopmentContainerIsRunning' loop..." action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:12:22Z" level=info msg="ticker event in 'waitUntilDevelopmentContainerIsRunning' loop..." action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:12:32Z" level=info msg="ticker event in 'waitUntilDevelopmentContainerIsRunning' loop..." action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:12:42Z" level=info msg="ticker event in 'waitUntilDevelopmentContainerIsRunning' loop..." action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:12:52Z" level=info msg="ticker event in 'waitUntilDevelopmentContainerIsRunning' loop..." action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:12:59Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:12:59Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:12:59Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
...
time="2024-02-10T15:13:02Z" level=info msg="ticker event in 'waitUntilDevelopmentContainerIsRunning' loop..." action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:02Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:02Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:02Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:02Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:02Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
...
4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:32Z" level=info msg="ticker event in 'waitUntilDevelopmentContainerIsRunning' loop..." action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:32Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:32Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:32Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:32Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:32Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:32Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:32Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:33Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:33Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:33Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:33Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:33Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:33Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:33Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:33Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:33Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:34Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:34Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:34Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:34Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:34Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:34Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:34Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:34Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:35Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:35Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:35Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
...

Steve · February 11, 2024, 10:04am

I’ve isolated and dug deeper and the problem is with Kubernetes chown every file in the volume on attachment.

Adding the following to the deployment yaml stops it doing a chown

      securityContext:
        fsGroup: 0     # Need to change all deployments to not use root!!
        fsGroupChangePolicy: "OnRootMismatch"

Topic		Replies	Views
Dev containers are slow Support	2	370	June 22, 2022
Syncthing cant find file that is container but not on server Support	3	325	May 9, 2023
Infinite loading for launch dev environment Support	10	346	October 14, 2022
Improve build performance by persisting the Buildkit cache layers message Okteto Self-Hosted	5	263	August 3, 2023
Okteto deploy and mounted volumes errors Support	0	445	May 27, 2022

externalVolumes with lots of files take forever to deploy

Related Topics