Installing Okteto helm chart cause all outbound connections to be routed to the Ingress

Okteto Self-Hosted
1

I’ve installed a proof-of-concept Okteto installation in a test EKS cluster at AWS. There is nothing else installed in the cluster, other than a few basics like aws-load-balancer-controller etc.

When the helm chart is installed, all outgoing connections from pods running in the cluster (in ANY namespace) resolve to the IP of the load balancer provisioned for the Ingress.

Performing a regular DNS lookup from a pod (using dig, for example) returns the correct result. But if I attempt to actually connect to that same domain name (using, for example, curl or ping, or any other service like apt), it instead resolves to the ALB, which then results in an SSL error, because of course the load balancer’s certificate doesn’t cover whatever host we were actually trying to connect to.

Here is an example, from an alpine pod started in a different namespace on the cluster:

19:09:04 tcgshell:/ # dig google.com

; <<>> DiG 9.11.3-1ubuntu1.17-Ubuntu <<>> google.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 9023
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 4a18e346069058c6 (echoed)
;; QUESTION SECTION:
;google.com.                    IN      A

;; ANSWER SECTION:
google.com.             30      IN      A       142.251.45.110

;; Query time: 1 msec
;; SERVER: 172.20.0.10#53(172.20.0.10)
;; WHEN: Tue Aug 16 19:19:15 UTC 2022
;; MSG SIZE  rcvd: 77

19:19:15 tcgshell:/ # ping google.com
PING ab3f1587d9570454184096e4e937672e-133400573.us-east-1.elb.amazonaws.com (54.167.74.170) 56(84) bytes of data.
^C
--- ab3f1587d9570454184096e4e937672e-133400573.us-east-1.elb.amazonaws.com ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4095ms

19:19:25 tcgshell:/ # curl https://google.com/
curl: (51) SSL: no alternative certificate subject name matches target host name 'google.com'

Uninstalling the Okteto helm chart makes the problem go away, and outbound connections work as expected. Re-installing the helm chart causes the problem to reappear.

Any ideas? Thanks.

2

I resolved the issue – my VPC had a custom DHCP option set, which was adding the subdomain to the nameserver search list. This subdomain was the same one I set up as a wildcard DNS record for Okteto to operate under, so every DNS lookup was matching the wildcard DNS record, and thus resolving to the address of the load balancer.

Possible solutions are to remove the custom DHCP option set, or pick a different subnet for Okteto than the one assigned to the VPC as a whole. I went with the former, and the application now works correctly.

1 Like
3

Thanks for the update @dmccorma and I’m glad you have it working.