MongoDB SRV address dns issues

amoscatelli · April 3, 2023, 7:31am

Hi !
I can’t connect my okteto cluster to my mongodb atlas using SRV addresses :

2023-03-31 12:28:30.88 UTCchat-5bbc5746fc-9g6dzchat12:28:30 INFO traceId=, parentId=, spanId=, sampled= [or.mo.dr.cluster] (cluster-ClusterId{value='6426d1d0c11d2a13b94ccbaa', description='null'}-srv-quarkus.uvyzu7h.mongodb.net) Exception while resolving SRV records: com.mongodb.MongoConfigurationException: Failed looking up SRV record for '_mongodb._tcp.quarkus.uvyzu7h.mongodb.net'.
2023-03-31 12:28:30.88 UTCchat-5bbc5746fc-9g6dzchat at com.mongodb.internal.dns.DefaultDnsResolver.resolveHostFromSrvRecords(DefaultDnsResolver.java:92)
2023-03-31 12:28:30.88 UTCchat-5bbc5746fc-9g6dzchat at com.mongodb.internal.connection.DefaultDnsSrvRecordMonitor$DnsSrvRecordMonitorRunnable.run(DefaultDnsSrvRecordMonitor.java:80)
2023-03-31 12:28:30.88 UTCchat-5bbc5746fc-9g6dzchat at java.base@17.0.6/java.lang.Thread.run(Thread.java:833)
2023-03-31 12:28:30.88 UTCchat-5bbc5746fc-9g6dzchat at org.graalvm.nativeimage.builder/com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:775)
2023-03-31 12:28:30.88 UTCchat-5bbc5746fc-9g6dzchat at org.graalvm.nativeimage.builder/com.oracle.svm.core.posix.thread.PosixPlatformThreads.pthreadStartRoutine(PosixPlatformThreads.java:203)
2023-03-31 12:28:30.88 UTCchat-5bbc5746fc-9g6dzchatCaused by: com.mongodb.MongoConfigurationException: Unable to look up SRV record for host _mongodb._tcp.quarkus.uvyzu7h.mongodb.net
2023-03-31 12:28:30.88 UTCchat-5bbc5746fc-9g6dzchat at io.quarkus.mongodb.runtime.dns.MongoDnsClient.resolveSrvRequest(MongoDnsClient.java:152)
2023-03-31 12:28:30.88 UTCchat-5bbc5746fc-9g6dzchat at io.quarkus.mongodb.runtime.dns.MongoDnsClient.getResourceRecordData(MongoDnsClient.java:104)
2023-03-31 12:28:30.88 UTCchat-5bbc5746fc-9g6dzchat at com.mongodb.internal.dns.DefaultDnsResolver.resolveHostFromSrvRecords(DefaultDnsResolver.java:74)
2023-03-31 12:28:30.88 UTCchat-5bbc5746fc-9g6dzchat ... 4 more
2023-03-31 12:28:30.88 UTCchat-5bbc5746fc-9g6dzchatCaused by: io.smallrye.mutiny.TimeoutException
2023-03-31 12:28:30.88 UTCchat-5bbc5746fc-9g6dzchat at io.smallrye.mutiny.operators.uni.UniBlockingAwait.await(UniBlockingAwait.java:64)
2023-03-31 12:28:30.88 UTCchat-5bbc5746fc-9g6dzchat at io.smallrye.mutiny.groups.UniAwait.atMost(UniAwait.java:65)
2023-03-31 12:28:30.88 UTCchat-5bbc5746fc-9g6dzchat at io.quarkus.mongodb.runtime.dns.MongoDnsClient.resolveSrvRequest(MongoDnsClient.java:138)
2023-03-31 12:28:30.88 UTCchat-5bbc5746fc-9g6dzchat ... 6 more

Everything works when using non-SRV connection strings, indeed I already enabled all ip (0.0.0.0/0) from my atlas security panel, so I don’t need to list my IPs.

After chatting with Quarkus dev teams, this seems related to the DNS configuration of the kubernetes cluster as this already works in other K8s clusters, what do I need to do to have this work in Okteto ?

Thank you in advance

tere · April 4, 2023, 2:13pm

Hi @amoscatelli ,

we’ve gone though your case and it is working correctly with okteto cloud.

Could you check the settings you are using to set this property?

Thanks,

amoscatelli · April 13, 2023, 7:24am

Sorry I can’t understand your question. What property ?
Shouldn’t this work out of the box or do I need to configure something about this ?

Thank you in advance

amoscatelli · April 13, 2023, 9:04am

This is my kubernetes descriptor :

---
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    app.quarkus.io/build-timestamp: 2023-04-13 - 07:19:47 +0000
  labels:
    app.kubernetes.io/version: 1.0.0-SNAPSHOT
    app.kubernetes.io/name: chat
    app.kubernetes.io/managed-by: quarkus
  name: chat
spec:
  replicas: 3
  selector:
    matchLabels:
      app.kubernetes.io/version: 1.0.0-SNAPSHOT
      app.kubernetes.io/name: chat
  template:
    metadata:
      annotations:
        app.quarkus.io/build-timestamp: 2023-04-13 - 07:19:47 +0000
      labels:
        app.kubernetes.io/version: 1.0.0-SNAPSHOT
        app.kubernetes.io/name: chat
        app.kubernetes.io/managed-by: quarkus
    spec:
      containers:
        - env:
            - name: KUBERNETES_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: RABBITMQ_HOST
              valueFrom:
                configMapKeyRef:
                  key: host
                  name: cloudamqp
            - name: QUARKUS_MONGODB_CONNECTION_STRING
              valueFrom:
                secretKeyRef:
                  key: url
                  name: mongodb
            - name: MP_JWT_VERIFY_PUBLICKEY_LOCATION
              valueFrom:
                configMapKeyRef:
                  key: publickey-location
                  name: auth0
            - name: RABBITMQ_USERNAME
              valueFrom:
                secretKeyRef:
                  key: username
                  name: cloudamqp
            - name: MP_JWT_VERIFY_ISSUER
              valueFrom:
                configMapKeyRef:
                  key: issuer
                  name: auth0
            - name: QUARKUS_OPENTELEMETRY_TRACER_EXPORTER_OTLP_ENDPOINT
              valueFrom:
                configMapKeyRef:
                  key: endpoint
                  name: newrelic
            - name: QUARKUS_OPENTELEMETRY_TRACER_EXPORTER_OTLP_HEADERS
              valueFrom:
                secretKeyRef:
                  key: headers
                  name: newrelic
            - name: RABBITMQ_VIRTUAL_HOST
              valueFrom:
                configMapKeyRef:
                  key: virtual-host
                  name: cloudamqp
            - name: RABBITMQ_PASSWORD
              valueFrom:
                secretKeyRef:
                  key: password
                  name: cloudamqp
            - name: QUARKUS_MONGODB_TRACING_ENABLED
              value: "true"
            - name: SMALLRYE_JWT_PATH_GROUPS
              valueFrom:
                configMapKeyRef:
                  key: path-groups
                  name: auth0
            - name: JNOSQL_DOCUMENT_DATABASE
              valueFrom:
                configMapKeyRef:
                  key: database
                  name: mongodb
          envFrom:
            - secretRef:
                name: mongodb
          image: registry.cloud.okteto.net/amoscatelli/chat:1.0.0-SNAPSHOT
          imagePullPolicy: Always
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /chat/q/health/live
              port: 8080
              scheme: HTTP
            initialDelaySeconds: 5
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10
          name: chat
          ports:
            - containerPort: 8443
              name: https
              protocol: TCP
            - containerPort: 8080
              name: http
              protocol: TCP
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /chat/q/health/ready
              port: 8080
              scheme: HTTP
            initialDelaySeconds: 5
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10
          startupProbe:
            failureThreshold: 3
            httpGet:
              path: /chat/q/health/started
              port: 8080
              scheme: HTTP
            initialDelaySeconds: 5
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    app.quarkus.io/build-timestamp: 2023-04-13 - 07:19:47 +0000
  labels:
    app.kubernetes.io/name: chat
    app.kubernetes.io/version: 1.0.0-SNAPSHOT
    app.kubernetes.io/managed-by: quarkus
  name: chat
spec:
  rules:
    - host: amoscatelli.cloud.okteto.net
      http:
        paths:
          - backend:
              service:
                name: chat
                port:
                  name: http
            path: /chat
            pathType: Prefix
---
apiVersion: v1
kind: Service
metadata:
  annotations:
    app.quarkus.io/build-timestamp: 2023-04-13 - 07:19:47 +0000
  labels:
    app.kubernetes.io/name: chat
    app.kubernetes.io/version: 1.0.0-SNAPSHOT
    app.kubernetes.io/managed-by: quarkus
  name: chat
spec:
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: 8080
    - name: https
      port: 443
      protocol: TCP
      targetPort: 8443
  selector:
    app.kubernetes.io/name: chat
    app.kubernetes.io/version: 1.0.0-SNAPSHOT
  type: ClusterIP

tere · April 13, 2023, 1:28pm

Hi, by property i meant the setting that configures the SRV addresses.

Can you share your okteto manifest to have more context regarding the issue.

Let us look into it with the manifests you have and get back.

Regards,

amoscatelli · April 14, 2023, 12:04am

@tere I am sorry, I forgot to specify I am using okteto cloud.
I guess this is why I don’t have an okteto manifest.
I just downloaded my kubernetes credentials to deploy on my namespace “amoscatelli”

Is there any specific config for okteto cloud to enable support for SRV records ?

adripedriza · April 14, 2023, 9:43am

Hi @amoscatelli
We are still looking into your issue. We will get back to you. Thank you!

amoscatelli · April 14, 2023, 11:07am

Ok ! I’ll be waiting for you feedback then.
Thank you !

adripedriza · April 18, 2023, 8:56am

Hi @amoscatelli
I have tested a case similar to yours where an app running in Cloud consumes a mongo atlas database using SRV (voting-app-with-external-resources/server.js at main · okteto/voting-app-with-external-resources · GitHub) and I have been able to do it. Using the example GitHub - okteto/voting-app-with-external-resources: This is an example of how to configure and deploy a development environment that includes polyglot microservices, an AWS lambda function, and uses MongoDB Atlas for storage.. Can you confirm that this example works for you?

amoscatelli · April 19, 2023, 8:19am

I can’t test the specific example cause I don’t have any AWS account to fill :
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY

I’ll find a simpler example accessing MongoDB (maybe an UI of some kind) and I’ll try it and return to you

amoscatelli · April 19, 2023, 3:49pm

I installed an ubuntu helm chart and from inside my pod I can succesfully resolve the SRV records using the hostname command from @tere

Also I can successfully telnet mongodb 27017 ports of my atlas cluster members

I am not a kubernetes expert, but I think this excludes any issue with Okteto.
This issue must lie somewhere in Quarkus, but I dunno why it rises only inside my okteto/kubernetes deployment.

I really dunno what to do to troubleshoot more …

adripedriza · April 20, 2023, 9:12am

Sorry I can’t be of more help with your problem if there is anything we can do on our side let us know!

Best regards

Topic		Replies	Views
Cant connect to mongodb atlas Support	10	508	December 15, 2022
Rasa with mongodb Support	8	631	November 7, 2022
Docker-compose mongodb error Okteto Self-Hosted	2	583	April 6, 2023
Self-host DNS setup How Do I?	0	264	November 29, 2022

MongoDB SRV address dns issues

Related Topics