externalVolumes with lots of files take forever to deploy

Steve · February 10, 2024, 2:35pm

Hi

I’m using the latest version to deploy. If I’ve got no externalVolumes or an empty one it deploys straight away. If I’ve got an externalVolume with hundreds of thousands of files, it goes away for hours before sometimes deploying. Any ideas?

dev:
  test:
    autocreate: true
    image:  busybox
    forward:
      - 8080:8080
      - 9229:9229
    workdir: /usr/src/app
    sync:
      - .:/usr/src/app
    externalVolumes:
      - historic::/external
    resources:
      requests:
        cpu: "500m"
        memory: "1000Mi"
      limits:
        cpu: "1000m"
        memory: "2000Mi"
    initContainer:
      image: okteto/bin:1.4.4
      resources:
        requests:
          cpu: "500m"
          memory: "1000Mi"
        limits:
          cpu: "1000m"
          memory: "2000Mi"

Steve · February 10, 2024, 3:23pm

The okteto.log file contains

time="2024-02-10T14:31:32Z" level=info msg="ticker event in 'waitUntilDevelopmentContainerIsRunning' loop..." action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T14:31:42Z" level=info msg="ticker event in 'waitUntilDevelopmentContainerIsRunning' loop..." action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T14:31:52Z" level=info msg="ticker event in 'waitUntilDevelopmentContainerIsRunning' loop..." action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T14:32:02Z" level=info msg="ticker event in 'waitUntilDevelopmentContainerIsRunning' loop..." action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T14:32:12Z" level=info msg="ticker event in 'waitUntilDevelopmentContainerIsRunning' loop..." action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T14:32:22Z" level=info msg="ticker event in 'waitUntilDevelopmentContainerIsRunning' loop..." action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
...
659fbb115bd6 version=2.25.1
time="2024-02-10T15:11:02Z" level=info msg="ticker event in 'waitUntilDevelopmentContainerIsRunning' loop..." action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:11:12Z" level=info msg="ticker event in 'waitUntilDevelopmentContainerIsRunning' loop..." action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:11:22Z" level=info msg="ticker event in 'waitUntilDevelopmentContainerIsRunning' loop..." action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:11:32Z" level=info msg="ticker event in 'waitUntilDevelopmentContainerIsRunning' loop..." action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:11:42Z" level=info msg="ticker event in 'waitUntilDevelopmentContainerIsRunning' loop..." action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:11:52Z" level=info msg="ticker event in 'waitUntilDevelopmentContainerIsRunning' loop..." action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:12:02Z" level=info msg="ticker event in 'waitUntilDevelopmentContainerIsRunning' loop..." action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:12:12Z" level=info msg="ticker event in 'waitUntilDevelopmentContainerIsRunning' loop..." action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:12:22Z" level=info msg="ticker event in 'waitUntilDevelopmentContainerIsRunning' loop..." action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:12:32Z" level=info msg="ticker event in 'waitUntilDevelopmentContainerIsRunning' loop..." action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:12:42Z" level=info msg="ticker event in 'waitUntilDevelopmentContainerIsRunning' loop..." action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:12:52Z" level=info msg="ticker event in 'waitUntilDevelopmentContainerIsRunning' loop..." action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:12:59Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:12:59Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:12:59Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
...
time="2024-02-10T15:13:02Z" level=info msg="ticker event in 'waitUntilDevelopmentContainerIsRunning' loop..." action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:02Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:02Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:02Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:02Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:02Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
...
4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:32Z" level=info msg="ticker event in 'waitUntilDevelopmentContainerIsRunning' loop..." action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:32Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:32Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:32Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:32Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:32Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:32Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:32Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:33Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:33Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:33Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:33Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:33Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:33Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:33Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:33Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:33Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:34Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:34Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:34Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:34Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:34Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:34Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:34Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:34Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:35Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:35Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
time="2024-02-10T15:13:35Z" level=info msg="failed to cast event" action=906c1146-4839-443c-b47a-659fbb115bd6 version=2.25.1
...

Steve · February 11, 2024, 10:04am

I’ve isolated and dug deeper and the problem is with Kubernetes chown every file in the volume on attachment.

Adding the following to the deployment yaml stops it doing a chown

      securityContext:
        fsGroup: 0     # Need to change all deployments to not use root!!
        fsGroupChangePolicy: "OnRootMismatch"

Topic		Replies	Views
Dev containers are slow Help	2	539	June 22, 2022
Suggestions for working with an enormous repository? How Do I?	1	447	August 3, 2023
My dev containers are slow, how can I make them faster? How Do I?	1	1196	November 13, 2023
Okteto Hanging at Pulling Images Help	18	641	January 19, 2023
Okteto deploy and mounted volumes errors Help	0	677	May 27, 2022

externalVolumes with lots of files take forever to deploy

Related topics