Issues with okteto-api reading wildcard SSL cert

Hello everyone!

I have been playing with Okteto for a few days along with a coworker since meeting the team at KubeCon, and so far we like what we see, but are trying to pass a roadblock on setting up our product inside Okteto. While attempting to enable SSL, we are unable to get the okteto-api deployment to recognize the cert. I have been following the instructions found here and am attempting to deploy a letsencrypt cert that has already been validated and is ready to use. I have attempted deploying the cert and the Helm chart a few times to recheck my steps and here is where I am so far:

Starting with a predeployed Okteto deployment with a self-signed cert (the okteto-api won’t start otherwise), I validate the cert locally with openssl and create the standard tls secret:

$ ls
cert1.pem  chain1.pem  fullchain1.pem  privkey1.pem
$ openssl x509 -noout -subject -dates -ext subjectAltName -in ./cert1.pem
subject=CN = *.dev-prism.<fqdn>
notBefore=Nov  3 17:20:04 2022 GMT
notAfter=Feb  1 17:20:03 2023 GMT
X509v3 Subject Alternative Name: 
    DNS:*.dev-prism.<fqdn>, DNS:dev-prism.<fqdn>
$ k create secret tls dev-prism-cert --key ./privkey1.pem --cert ./cert1.pem --namespace okteto
secret/dev-prism-cert created
$ k describe secret dev-prism-cert
Name:         dev-prism-cert
Namespace:    okteto
Labels:       <none>
Annotations:  <none>


tls.key:  1704 bytes
tls.crt:  1911 bytes

I then updated the Helm chart values files to include the following:

  create: false
  name: dev-prism-cert

      default-ssl-certificate: "okteto/dev-prism-cert"

Redeploy was successful:

$ helm upgrade okteto okteto/okteto -f dev-utils/helm/postoffice/dev-prism/okteto-values.yml -n okteto
Release "okteto" has been upgraded. Happy Helming!
NAME: okteto
LAST DEPLOYED: Mon Nov  7 12:38:35 2022
STATUS: deployed

Inspecting the cluster shows the api rolling release failing:

$ k get po
NAME                                                  READY   STATUS             RESTARTS      AGE
okteto-api-596c98c444-46586                           0/1     CrashLoopBackOff   3 (38s ago)   88s
okteto-api-767f6c5f74-bnnb9                           1/1     Running            0             2d21h
okteto-autoscaler-5b5f786749-7bfjm                    1/1     Running            0             88s
okteto-buildkit-5b21767586-0                          1/1     Running            0             57s
okteto-daemon-ltgmn                                   1/1     Running            0             89s
okteto-daemon-wxtf6                                   1/1     Running            0             89s
okteto-daemon-zsqrp                                   1/1     Running            0             89s
okteto-destroy-all-checker-27797373-zg6gx             0/1     Completed          0             7m11s
okteto-destroy-all-checker-27797376-rk67f             0/1     Completed          0             4m11s
okteto-destroy-all-checker-27797379-hhp5t             0/1     Completed          0             71s
okteto-frontend-67cfff89cf-4qmzw                      1/1     Running            0             2d21h
okteto-frontend-67cfff89cf-mw99l                      1/1     Running            0             2d21h
okteto-ingress-nginx-controller-797cfb4b66-h7ls6      1/1     Running            0             68s
okteto-ingress-nginx-controller-797cfb4b66-m2m7x      1/1     Running            0             89s
okteto-ingress-nginx-defaultbackend-c54f6c679-6p74m   1/1     Running            0             2d21h
okteto-ingress-nginx-defaultbackend-c54f6c679-6txbm   1/1     Running            0             2d21h
okteto-installer-checker-27797370-fc6jw               0/1     Completed          0             10m
okteto-installer-checker-27797375-zrjcx               0/1     Completed          0             5m11s
okteto-installer-checker-27797380-6vghq               0/1     Completed          0             11s
okteto-migration-a2643dd5b759268e-ljg6s               0/1     Error              0             88s
okteto-migration-a2643dd5b759268e-mjhtf               0/1     Error              0             83s
okteto-migration-a2643dd5b759268e-t2g2v               0/1     Completed          0             73s
okteto-mutation-webhook-74cfb97cbf-8j2m2              1/1     Running            0             88s
okteto-mutation-webhook-74cfb97cbf-c74sw              1/1     Running            0             77s
okteto-registry-5b4c9d89bb-x5cjc                      1/1     Running            0             88s
okteto-reloader-754dd844b6-82lm9                      1/1     Running            1             2d21h
okteto-telemetry-27793440-8w5sw                       0/1     Completed          0             2d17h
okteto-telemetry-27794880-g9r59                       0/1     Completed          0             41h
okteto-telemetry-27796320-n7kdc                       0/1     Completed          0             17h

The logs show that it is unable to read the cert as a valid:

$ k logs okteto-api-596c98c444-46586 
{"timestamp":"2022-11-07T17:40:25Z","message":"Starting api service...","severity":"INFO","context":{"data":{"":"okteto-api-596c98c444-46586"}}}
{"timestamp":"2022-11-07T17:40:25Z","message":"using no-op tracer","severity":"INFO","context":{"data":{"":"okteto-api-596c98c444-46586"}}}
{"timestamp":"2022-11-07T17:40:25Z","message":"loading buildkit certificate","severity":"INFO","context":{"data":{"":"okteto-api-596c98c444-46586"}}}
{"timestamp":"2022-11-07T17:40:25Z","serviceContext":{"service":"okteto-enterprise-backend"},"message":"couldn't find a valid cert: unable to decode private key PEM: unable to get PrivateKey from PEM type: PRIVATE KEY","severity":"CRITICAL","context":{"data":{"":"okteto-api-596c98c444-46586"},"reportLocation":{"filePath":"","lineNumber":52,"functionName":"Init"}}}

However, the ingress-nginx deployment is able to read the cert successfully:

$ k describe ingress okteto
Name:             okteto
Namespace:        okteto
Address:          <redacted>
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
  dev-prism-cert terminates<fqdn>

The configuration shows it is using the cert with the default-ssl-certificate arg:

$ k describe po okteto-ingress-nginx-controller-797cfb4b66-h7ls6 
Name:         okteto-ingress-nginx-controller-797cfb4b66-h7ls6
Namespace:    okteto
Priority:     0
Node:         ip-10-0-1-173.ec2.internal/
Start Time:   Mon, 07 Nov 2022 12:39:03 -0500
Annotations: eks.privileged
Status:       Running
Controlled By:  ReplicaSet/okteto-ingress-nginx-controller-797cfb4b66
    Container ID:  docker://87e64bff3947e4661bba09d1ad6924375d6f58190363fd423cef64dd6cb6d0aa
    Image ID:      docker-pullable://
    Ports:         80/TCP, 443/TCP, 1234/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP
    State:          Running
      Started:      Mon, 07 Nov 2022 12:39:04 -0500
    Ready:          True

And when I view the site, the insecure https splash page no longer appears and shows a valid cert:

I’m not sure what I am missing, as I think I followed all the steps correctly, but it’s possible I missed something in the documentation, or simply typo’d something. Any assistance would be greatly appreciated!


@pmarks how did you create the initial certificates?

Hi @ramiro! Since using cert-manager would require setting up cross-account IAM roles that I currently don’t have permissions to do, I opted to create the cert by hand using certbot and performed the HTTP-01 and DNS-01 challenge by hand. We will look to set up cert-manager for a more permanent solution with this to get us through our POC.

That makes sense, that’s a great way to get around that issue! Looking at the instructions for certbot on this repo (I’m assuming you followed a similar path, please let me know if you didn’t), could you try re-importing the certificate but using fullchain1.pem as the value of --key?

Hello again,

Yes, I followed a similar path, except I ran certbot locally and used a readily available webserver (instead of containers as per the docs) to perform the HTTP-01 challenge. I also didn’t specify the challenge I wanted to do, so I had to perform both an HTTP and a DNS challenge for the cert to be valid.

I did attempt to add the cert with the full chain pemfile as you recommended, but received the same error:

$ k create secret tls dev-prism-cert --key=privkey1.pem --cert=fullchain1.pem --namespace okteto
secret/dev-prism-cert created

After a fresh helm install:

$ k logs okteto-api-596c98c444-9fr2m 
{"timestamp":"2022-11-08T19:17:06Z","message":"Starting api service...","severity":"INFO","context":{"data":{"":"okteto-api-596c98c444-9fr2m"}}}
{"timestamp":"2022-11-08T19:17:06Z","message":"using no-op tracer","severity":"INFO","context":{"data":{"":"okteto-api-596c98c444-9fr2m"}}}
{"timestamp":"2022-11-08T19:17:06Z","message":"loading buildkit certificate","severity":"INFO","context":{"data":{"":"okteto-api-596c98c444-9fr2m"}}}
{"timestamp":"2022-11-08T19:17:06Z","serviceContext":{"service":"okteto-enterprise-backend"},"message":"couldn't find a valid cert: unable to decode private key PEM: unable to get PrivateKey from PEM type: PRIVATE KEY","severity":"CRITICAL","context":{"data":{"":"okteto-api-596c98c444-9fr2m"},"reportLocation":{"filePath":"","lineNumber":52,"functionName":"Init"}}}

I originally tried an update but saw no change of behavior, so I attempted a fresh install to ensure it was trying the latest version of the cert.

Hi @pmarks,

I bet your private key is in PKCS8 format, which is not currently supported by Okteto components.

The following PEM header matches PKCS8:


The following PEM header matches PKCS1:


To convert your current PKCS8 private key into PKCS1, use the following command. Replace cert-pkcs8.pem with the filename of your current private key. Replace cert-pkcs1.pem with the desired filename for your PKCS1 private key.

openssl rsa -in cert-pkcs8.pem -out cert-pkcs1.pem
1 Like

@provecho That did the trick! The command that worked for me was slightly different than the one you gave:

$ openssl pkcs8 -in cert-pkcs8.pem -traditional -nocrypt -out cert-pkcs1.pem

Otherwise, it was not changing the format as expected.

Once I updated the cert and reinstalled Okteto, everything worked as expected! I can now verify the api is happy and okteto context works without the insecure flag. Thanks very much to you and to @ramiro for your assistance, as we are now unblocked and can work on our proof-of-concept further! Once we get a bit of time to play around and gather our questions, we will be reaching back out to talk further! Thanks again to you both!



Great! We’ll update our docs to be clearer about the supported certificate formats.