Issues with okteto-api reading wildcard SSL cert

Okteto Self-Hosted
1

Hello everyone!

I have been playing with Okteto for a few days along with a coworker since meeting the team at KubeCon, and so far we like what we see, but are trying to pass a roadblock on setting up our product inside Okteto. While attempting to enable SSL, we are unable to get the okteto-api deployment to recognize the cert. I have been following the instructions found here and am attempting to deploy a letsencrypt cert that has already been validated and is ready to use. I have attempted deploying the cert and the Helm chart a few times to recheck my steps and here is where I am so far:

Starting with a predeployed Okteto deployment with a self-signed cert (the okteto-api won’t start otherwise), I validate the cert locally with openssl and create the standard tls secret:

$ ls
cert1.pem  chain1.pem  fullchain1.pem  privkey1.pem
$ openssl x509 -noout -subject -dates -ext subjectAltName -in ./cert1.pem
subject=CN = *.dev-prism.<fqdn>
notBefore=Nov  3 17:20:04 2022 GMT
notAfter=Feb  1 17:20:03 2023 GMT
X509v3 Subject Alternative Name: 
    DNS:*.dev-prism.<fqdn>, DNS:dev-prism.<fqdn>
$ k create secret tls dev-prism-cert --key ./privkey1.pem --cert ./cert1.pem --namespace okteto
secret/dev-prism-cert created
$ k describe secret dev-prism-cert
Name:         dev-prism-cert
Namespace:    okteto
Labels:       <none>
Annotations:  <none>

Type:  kubernetes.io/tls

Data
====
tls.key:  1704 bytes
tls.crt:  1911 bytes

I then updated the Helm chart values files to include the following:

wildcardCertificate:
  create: false
  name: dev-prism-cert

ingress-nginx:
  controller:
    extraArgs:
      default-ssl-certificate: "okteto/dev-prism-cert"

Redeploy was successful:

$ helm upgrade okteto okteto/okteto -f dev-utils/helm/postoffice/dev-prism/okteto-values.yml -n okteto
Release "okteto" has been upgraded. Happy Helming!
NAME: okteto
LAST DEPLOYED: Mon Nov  7 12:38:35 2022
NAMESPACE: okteto
STATUS: deployed
REVISION: 2
TEST SUITE: None
NOTES:
<truncated>

Inspecting the cluster shows the api rolling release failing:

$ k get po
NAME                                                  READY   STATUS             RESTARTS      AGE
okteto-api-596c98c444-46586                           0/1     CrashLoopBackOff   3 (38s ago)   88s
okteto-api-767f6c5f74-bnnb9                           1/1     Running            0             2d21h
okteto-autoscaler-5b5f786749-7bfjm                    1/1     Running            0             88s
okteto-buildkit-5b21767586-0                          1/1     Running            0             57s
okteto-daemon-ltgmn                                   1/1     Running            0             89s
okteto-daemon-wxtf6                                   1/1     Running            0             89s
okteto-daemon-zsqrp                                   1/1     Running            0             89s
okteto-destroy-all-checker-27797373-zg6gx             0/1     Completed          0             7m11s
okteto-destroy-all-checker-27797376-rk67f             0/1     Completed          0             4m11s
okteto-destroy-all-checker-27797379-hhp5t             0/1     Completed          0             71s
okteto-frontend-67cfff89cf-4qmzw                      1/1     Running            0             2d21h
okteto-frontend-67cfff89cf-mw99l                      1/1     Running            0             2d21h
okteto-ingress-nginx-controller-797cfb4b66-h7ls6      1/1     Running            0             68s
okteto-ingress-nginx-controller-797cfb4b66-m2m7x      1/1     Running            0             89s
okteto-ingress-nginx-defaultbackend-c54f6c679-6p74m   1/1     Running            0             2d21h
okteto-ingress-nginx-defaultbackend-c54f6c679-6txbm   1/1     Running            0             2d21h
okteto-installer-checker-27797370-fc6jw               0/1     Completed          0             10m
okteto-installer-checker-27797375-zrjcx               0/1     Completed          0             5m11s
okteto-installer-checker-27797380-6vghq               0/1     Completed          0             11s
okteto-migration-a2643dd5b759268e-ljg6s               0/1     Error              0             88s
okteto-migration-a2643dd5b759268e-mjhtf               0/1     Error              0             83s
okteto-migration-a2643dd5b759268e-t2g2v               0/1     Completed          0             73s
okteto-mutation-webhook-74cfb97cbf-8j2m2              1/1     Running            0             88s
okteto-mutation-webhook-74cfb97cbf-c74sw              1/1     Running            0             77s
okteto-registry-5b4c9d89bb-x5cjc                      1/1     Running            0             88s
okteto-reloader-754dd844b6-82lm9                      1/1     Running            1             2d21h
okteto-telemetry-27793440-8w5sw                       0/1     Completed          0             2d17h
okteto-telemetry-27794880-g9r59                       0/1     Completed          0             41h
okteto-telemetry-27796320-n7kdc                       0/1     Completed          0             17h

The logs show that it is unable to read the cert as a valid:


$ k logs okteto-api-596c98c444-46586 
{"timestamp":"2022-11-07T17:40:25Z","message":"Starting api service...","severity":"INFO","context":{"data":{"instance.id":"okteto-api-596c98c444-46586"}}}
{"timestamp":"2022-11-07T17:40:25Z","message":"using no-op tracer","severity":"INFO","context":{"data":{"instance.id":"okteto-api-596c98c444-46586"}}}
{"timestamp":"2022-11-07T17:40:25Z","message":"loading buildkit certificate","severity":"INFO","context":{"data":{"instance.id":"okteto-api-596c98c444-46586"}}}
{"timestamp":"2022-11-07T17:40:25Z","serviceContext":{"service":"okteto-enterprise-backend"},"message":"couldn't find a valid cert: unable to decode private key PEM: unable to get PrivateKey from PEM type: PRIVATE KEY","severity":"CRITICAL","context":{"data":{"instance.id":"okteto-api-596c98c444-46586"},"reportLocation":{"filePath":"github.com/okteto/app/backend/pkg/registry/registry.go","lineNumber":52,"functionName":"Init"}}}

However, the ingress-nginx deployment is able to read the cert successfully:


$ k describe ingress okteto
Name:             okteto
Labels:           app.kubernetes.io/component=api
                  app.kubernetes.io/instance=okteto
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=okteto
                  app.kubernetes.io/part-of=okteto
                  helm.sh/chart=okteto-1.0.0
Namespace:        okteto
Address:          <redacted>.us-east-1.elb.amazonaws.com
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
TLS:
  dev-prism-cert terminates okteto.dev-prism.<fqdn>

The configuration shows it is using the cert with the default-ssl-certificate arg:

$ k describe po okteto-ingress-nginx-controller-797cfb4b66-h7ls6 
Name:         okteto-ingress-nginx-controller-797cfb4b66-h7ls6
Namespace:    okteto
Priority:     0
Node:         ip-10-0-1-173.ec2.internal/10.0.1.173
Start Time:   Mon, 07 Nov 2022 12:39:03 -0500
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=okteto
              app.kubernetes.io/name=ingress-nginx
              pod-template-hash=797cfb4b66
Annotations:  kubernetes.io/psp: eks.privileged
Status:       Running
IP:           10.0.1.210
IPs:
  IP:           10.0.1.210
Controlled By:  ReplicaSet/okteto-ingress-nginx-controller-797cfb4b66
Containers:
  controller:
    Container ID:  docker://87e64bff3947e4661bba09d1ad6924375d6f58190363fd423cef64dd6cb6d0aa
    Image:         registry.k8s.io/ingress-nginx/controller:v1.3.0@sha256:d1707ca76d3b044ab8a28277a2466a02100ee9f58a86af1535a3edf9323ea1b5
    Image ID:      docker-pullable://registry.k8s.io/ingress-nginx/controller@sha256:d1707ca76d3b044ab8a28277a2466a02100ee9f58a86af1535a3edf9323ea1b5
    Ports:         80/TCP, 443/TCP, 1234/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP
    Args:
      /nginx-ingress-controller
      --default-backend-service=$(POD_NAMESPACE)/okteto-ingress-nginx-defaultbackend
      --publish-service=$(POD_NAMESPACE)/okteto-ingress-nginx-controller
      --election-id=ingress-controller-leader
      --controller-class=k8s.io/ingress-nginx
      --ingress-class=nginx
      --configmap=$(POD_NAMESPACE)/okteto-ingress-nginx-controller
      --tcp-services-configmap=$(POD_NAMESPACE)/okteto-ingress-nginx-tcp
      --default-ssl-certificate=okteto/dev-prism-cert
      --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services-configmap
    State:          Running
      Started:      Mon, 07 Nov 2022 12:39:04 -0500
    Ready:          True
<truncated>

And when I view the site, the insecure https splash page no longer appears and shows a valid cert:

okteto-letsencrypt
okteto-letsencrypt667×972 96.3 KB

I’m not sure what I am missing, as I think I followed all the steps correctly, but it’s possible I missed something in the documentation, or simply typo’d something. Any assistance would be greatly appreciated!

Peter

2

@pmarks how did you create the initial certificates?

3

Hi @ramiro! Since using cert-manager would require setting up cross-account IAM roles that I currently don’t have permissions to do, I opted to create the cert by hand using certbot and performed the HTTP-01 and DNS-01 challenge by hand. We will look to set up cert-manager for a more permanent solution with this to get us through our POC.

4

That makes sense, that’s a great way to get around that issue! Looking at the instructions for certbot on this repo (I’m assuming you followed a similar path, please let me know if you didn’t), could you try re-importing the certificate but using fullchain1.pem as the value of --key?

5

Hello again,

Yes, I followed a similar path, except I ran certbot locally and used a readily available webserver (instead of containers as per the docs) to perform the HTTP-01 challenge. I also didn’t specify the challenge I wanted to do, so I had to perform both an HTTP and a DNS challenge for the cert to be valid.

I did attempt to add the cert with the full chain pemfile as you recommended, but received the same error:

$ k create secret tls dev-prism-cert --key=privkey1.pem --cert=fullchain1.pem --namespace okteto
secret/dev-prism-cert created

After a fresh helm install:

$ k logs okteto-api-596c98c444-9fr2m 
{"timestamp":"2022-11-08T19:17:06Z","message":"Starting api service...","severity":"INFO","context":{"data":{"instance.id":"okteto-api-596c98c444-9fr2m"}}}
{"timestamp":"2022-11-08T19:17:06Z","message":"using no-op tracer","severity":"INFO","context":{"data":{"instance.id":"okteto-api-596c98c444-9fr2m"}}}
{"timestamp":"2022-11-08T19:17:06Z","message":"loading buildkit certificate","severity":"INFO","context":{"data":{"instance.id":"okteto-api-596c98c444-9fr2m"}}}
{"timestamp":"2022-11-08T19:17:06Z","serviceContext":{"service":"okteto-enterprise-backend"},"message":"couldn't find a valid cert: unable to decode private key PEM: unable to get PrivateKey from PEM type: PRIVATE KEY","severity":"CRITICAL","context":{"data":{"instance.id":"okteto-api-596c98c444-9fr2m"},"reportLocation":{"filePath":"github.com/okteto/app/backend/pkg/registry/registry.go","lineNumber":52,"functionName":"Init"}}}

I originally tried an update but saw no change of behavior, so I attempted a fresh install to ensure it was trying the latest version of the cert.

6

Hi @pmarks,

I bet your private key is in PKCS8 format, which is not currently supported by Okteto components.

The following PEM header matches PKCS8:

-----BEGIN PRIVATE KEY-----

The following PEM header matches PKCS1:

-----BEGIN RSA PRIVATE KEY-----

To convert your current PKCS8 private key into PKCS1, use the following command. Replace cert-pkcs8.pem with the filename of your current private key. Replace cert-pkcs1.pem with the desired filename for your PKCS1 private key.

openssl rsa -in cert-pkcs8.pem -out cert-pkcs1.pem
7

@provecho That did the trick! The command that worked for me was slightly different than the one you gave:

$ openssl pkcs8 -in cert-pkcs8.pem -traditional -nocrypt -out cert-pkcs1.pem

Otherwise, it was not changing the format as expected.

Once I updated the cert and reinstalled Okteto, everything worked as expected! I can now verify the api is happy and okteto context works without the insecure flag. Thanks very much to you and to @ramiro for your assistance, as we are now unblocked and can work on our proof-of-concept further! Once we get a bit of time to play around and gather our questions, we will be reaching back out to talk further! Thanks again to you both!

Peter

8

Great! We’ll update our docs to be clearer about the supported certificate formats.