Syncthing cant find file that is container but not on server

Help
1

My scenario: I have a kubernetes cluster and I’m using okteto which has syncthing installed. I run okteto up and it starts a pod in the cluster. Syncthing seems to be working fine at this point as I can see my directory and all the folders. When I run a job, all the files are created if I look at the pod within the cluster itself. However, if I look at the file directly on the server running syncthing, I have alot of .tmp files that dont appear in the pod.
3 12:31 file.txt-714
3 12:31 file.txt-723
3 12:31 file.txt-750
3 12:31 file.txt-794
3 12:31 file.txt-80
3 12:31 file.txt-94
3 12:31 .syncthing.file.txt-10.tmp
3 12:31 .syncthing.file.txt-115.tmp
3 12:31 .syncthing.file.txt-134.tmp
3 12:31 .syncthing.file.txt-158.tmp
3 12:31 .syncthing.file.txt-192.tmp
3 12:31 .syncthing.file.txt-232.tmp

At this point, my okteto into the pod has frozen and I’ve lost access: While looking at the okteto logs it says syncthing has failed:
events=FolderSummary&limit=1&since=0&timeout=0": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" action=1ea7e90c-f1a0-49a1-b8a0-d20ba5c81258 version=1.15.8
time=“2023-05-03T12:41:44-06:00” level=info msg=“retrying syncthing call[rest/events] local=false: failed to call syncthing [rest/events]: Get "http://localhost:33220/rest/events?events=FolderSummary&limit=1&since=0&timeout=0\”: net/http: request canceled (Client.Timeout exceeded while awaiting headers)" action=1ea7e90c-f1a0-49a1-b8a0-d20ba5c81258 version=1.15.8
time=“2023-05-03T12:41:49-06:00” level=info msg=“error getting events: failed to call syncthing [rest/events]: Get "http://localhost:33220/rest/events?events=FolderSummary&limit=1&since=0&timeout=0\”: net/http: request canceled (Client.Timeout exceeded while awaiting headers)" action=1ea7e90c-f1a0-49a1-b8a0-d20ba5c81258 version=1.15.8
time=“2023-05-03T12:42:29-06:00” level=info msg=“syncthing pull error local=true: syncing: no connected device has the required version of this file\n” action=1ea7e90c-f1a0-49a1-b8a0-d20ba5c81258 version=1.15.8
time=“2023-05-03T12:42:29-06:00” level=info msg=“syncthing error local=true retry 4: test-output/file.txt-1: syncing: no connected device has the required version of this file\n” action=1ea7e90c-f1a0-49a1-b8a0-d20ba5c81258 version=1.15.8
time=“2023-05-03T12:42:29-06:00” level=info msg=“syncthing monitor error, sending disconnect signal: test-output/file.txt-1: syncing: no connected device has the required version of this file\n” action=1ea7e90c-f1a0-49a1-b8a0-d20ba5c81258 version=1.15.8
time=“2023-05-03T12:42:29-06:00” level=info msg=“starting shutdown sequence” action=1ea7e90c-f1a0-49a1-b8a0-d20ba5c81258 version=1.15.8
time=“2023-05-03T12:42:29-06:00” level=info msg=“sent cancellation signal” action=1ea7e90c-f1a0-49a1-b8a0-d20ba5c81258 version=1.15.8
time=“2023-05-03T12:42:29-06:00” level=info msg=“stopping syncthing” action=1ea7e90c-f1a0-49a1-b8a0-d20ba5c81258 version=1.15.8
time=“2023-05-03T12:42:29-06:00” level=debug msg=“call to up.applyToApp cancelled: {}” action=1ea7e90c-f1a0-49a1-b8a0-d20ba5c81258 version=1.15.8
time=“2023-05-03T12:42:29-06:00” level=info msg=“ssh client for exec closed” action=1ea7e90c-f1a0-49a1-b8a0-d20ba5c81258 version=1.15.8
time=“2023-05-03T12:42:29-06:00” level=info msg=“command failed: wait: remote command exited without exit status or exit signal” action=1ea7e90c-f1a0-49a1-b8a0-d20ba5c81258 version=1.15.8
time=“2023-05-03T12:42:29-06:00” level=info msg=“terminal restored” action=1ea7e90c-f1a0-49a1-b8a0-d20ba5c81258 version=1.15.8
time=“2023-05-03T12:42:29-06:00” level=info msg=“ssh forward localhost:33220->0.0.0.0:8384 → done” action=1ea7e90c-f1a0-49a1-b8a0-d20ba5c81258 version=1.15.8
time=“2023-05-03T12:42:29-06:00” level=info msg=“ssh forward localhost:40654->0.0.0.0:22000 → done” action=1ea7e90c-f1a0-49a1-b8a0-d20ba5c81258 version=1.15.8
time=“2023-05-03T12:42:29-06:00” level=info msg=“syncthing ping error 0” action=1ea7e90c-f1a0-49a1-b8a0-d20ba5c81258 version=1.15.8
time=“2023-05-03T12:42:29-06:00” level=info msg=“terminating syncthing 167130 without wait” action=1ea7e90c-f1a0-49a1-b8a0-d20ba5c81258 version=1.15.8
time=“2023-05-03T12:42:29-06:00” level=info msg=“terminated syncthing 167130 without wait” action=1ea7e90c-f1a0-49a1-b8a0-d20ba5c81258 version=1.15.8
time=“2023-05-03T12:42:29-06:00” level=info msg=“stopping forwarders” action=1ea7e90c-f1a0-49a1-b8a0-d20ba5c81258 version=1.15.8
time=“2023-05-03T12:42:29-06:00” level=info msg=“stopped k8s forwarder” action=1ea7e90c-f1a0-49a1-b8a0-d20ba5c81258 version=1.15.8
time=“2023-05-03T12:42:29-06:00” level=info msg=“stopped SSH forward manager” action=1ea7e90c-f1a0-49a1-b8a0-d20ba5c81258 version=1.15.8
time=“2023-05-03T12:42:29-06:00” level=info msg=“completed shutdown sequence” action=1ea7e90c-f1a0-49a1-b8a0-d20ba5c81258 version=1.15.8
time=“2023-05-03T12:42:29-06:00” level=info msg=“activate failed with: test-output/file.txt-1: syncing: no connected device has the required version of this file\n” action=1ea7e90c-f1a0-49a1-b8a0-d20ba5c81258 ve

My syncthing log shows errors which seem to be indicating they cant find the files because of the .tmp
[ABKAV] 12:40:20 INFO: Listen (BEP/quic): listen udp 0.0.0.0:22000: bind: address already in use
[ABKAV] 12:40:20 INFO: listenerSupervisor@quic://0.0.0.0:22000: service quic://0.0.0.0:22000 failed: listen udp 0.0.0.0:22000: bind: address already in use
[ABKAV] 12:40:32 INFO: Puller (folder “1” (okteto-1), item “test-output/file.txt-573”): syncing: no connected device has the required version of this file
[ABKAV] 12:40:32 INFO: Puller (folder “1” (okteto-1), item “test-output/file.txt-118”): syncing: no connected device has the required version of this file
[ABKAV] 12:40:32 INFO: Puller (folder “1” (okteto-1), item “test-output/file.txt-815”): syncing: no connected device has the required version of this file
[ABKAV] 12:40:32 INFO: Puller (folder “1” (okteto-1), item “test-output/file.txt-737”): syncing: no connected device has the required version of this file
[ABKAV] 12:40:32 INFO: Puller (folder “1” (okteto-1), item “test-output/file.txt-387”): syncing: no connected device has the required version of this file
[ABKAV] 12:40:32 INFO: Puller (folder “1” (okteto-1), item “test-output/file.txt-350”): syncing: no connected device has the required version of this file
[ABKAV] 12:40:32 INFO: Puller (folder “1” (okteto-1), item “test-output/file.txt-352”): syncing: no connected device has the required version of this file

My container eventually dies saying it cant find a specific file which is in the container, but not on the server file it is synced to but it is in the container itself.

o-57b7g7slz:~$ x Test-output/file.txt-1: syncing: no connected device has the required version of this file

2

Hi!
Does this occur regularly?
This part of the log makes me think this could potentially be connectivity issues or possibly it is hitting a firewall that prevents its connection to the destination:
context deadline exceeded

3

Hi Jona,

Thanks for responding. It happens regularly as in daily yes. I’ve been running tests and it will sometimes pass fine like 2-3 times in a row and then fail several times in a row. It makes it really hard to troubleshoot the issue. Can’t find much online either. The file that it will complain about, for example, file.txt-0, won’t even be on the server in normal form or .tmp form. But it is on the container if I’m directly exec’d into the container. But it won’t be on the actual server if I’m on there directly and look. But this time, the my container has frozen so I cant look in okteto.
Could it be a quic/relay thing? Just something I saw online saying it could cause a slow connection. But oddly I don’t even see a .tmp for it.
I’m not sure this is the issue, just something else I noticed

4

When your application is running with okteto up and syncthing fails, are you able to run in a different terminal window the command okteto status <your container name> --info?

That should give you information for the syncthing UI. You should receive a localhost URL that you can open on your browser, type the provided username and password, and you should be logged in to see the Syncthing UI. Does the UI give any additional error information?