Registry with Encrypted AWS S3 bucket

Support
1

I am currently following the guide published here on how to connect okteto to S3 using an IAM Role. I’ve hooked up several service accounts to IAM roles in other projects, but this one seems to be failing. I’m getting a continual restart with no clear error message, and the troubleshooting docs suggest this is because it cannot connect to s3.

We cannot use IAM Users in our org, so IAM roles are our only option. Whilst the policy I have created is identical to the recomended one, it differs in one respect, which is I’ve included an extra statement to allow for the decryption using our KMS key. All our S3 buckets are required to have Server Side KMS encryption as per another organisation security policy. I suspect this is why okteto is unable to connect to it. The IAM policy should be correct and I’ve done similar with other apps, so I’d appreciate if someone could confirm that this is supported and help me to debug what might be going wrong with our setup.

The extra statement in the IAM policy looks like the following. The rest of it is identical to the recomended.

{
  "Action": [
    "kms:Decrypt",
    "kms:GenerateDataKey"
  ],
  "Effect": "Allow",
  "Resource": "*"
}
2

@cheer.b95 could you share the entire definition of your IAM Role?

Also, could you share the logs of the okteto-registry pods?

3 
cloud:
  provider:
    aws:
      iam:
        enabled: false

registry:
  serviceAccount:
    create: true
    annotations:
      eks.amazonaws.com/role-arn: 'arn:aws:iam::${AWS_ACCOUNT_ID}:role/${ROLE_NAME}'

The above is my config, and I can confirm the service account has the correct annotation and the pod has the correct AWS env variables, namely the AWS_WEB_IDENTITY_TOKEN_FILE.

the IAM Role’s policy is as follows. It’s overly permissive for testing:

{
"Action":"s3:*",
"Effect":"Allow",
"Resource":"*",
"Sid":""
},
{
"Action":"kms:*",
"Effect":"Allow",
"Resource":"*",
"Sid":""
}

as far as logs go, we see this on loop before the pod just disappears.

{
"go.version":"go1.16.15",
"instance.id":<SOME_GUID>,
"level":"debug",
"msg":"s3aws.Stat(\"/\")",
"service":"registry",
"time": <THE_TIME>,
"trace.duration": 901800,
"trace.file":"/go/src/github.com/docker/distributions/registry/storage/driver/base/base.go",
"trace.func":"github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat",
"trace.id":<SOME_GUID>,
"trace.line":155,
"version":"v2.8.1+unkown"
}

The only thing I can think is that okteto is not able to handle server side encryption, or its not reading the AWS_WEB_IDENTITY_TOKEN_FILE environment variable.

4

Would it be possible to test your setup without server-side encryption (on a different bucket) just to discard that? We’ll also do some testing around this internally.

5

Sadly not. Server side encryption is automatically applied to all our s3 buckets. We cannot disable it.

6

@cheer.b95 IAM Roles are not supported yet by the docker registry, they had a regression there and there is an open PR to fix it. In the meantime, you might use IAM users, or use a single registry replica and file system storage for now.

7

Another security policy prevents us from creating IAM users. We have to use roles. Hopefully this can be fixed quickly.

We’ll look into alternative storage for now but We would obviously prefer s3 if possible.

8

Is it possible to have a link to the helm chart code? I’ve not been able to find it in Github.

9

I can confirm switching to ‘byo’ and away from s3 temporarily fixed the issue. We’d like to switch to s3 and IAM roles at some point when this issue is resolved.

10

The helm chart code is not available through GitHub. However, you can inspect all helm charts by using helm pull, Helm | Helm Pull

11

I was curious if this had been fixed yet?

12

Hi @cheer.b95!

This should be fixable now by setting up an envvar for the registry. More official documentation from distribution/distribution here

In your helm Values.yaml you should add the extraEnv to the registry

registry:
  extraEnv:
  - name: REGISTRY_STORAGE_S3_ENCRYPT
    value: true

This should fix the issue you have with the encryption

Please let us know if you encounter another issue :smiley:

Hope this helps!

1 Like