A specific pod cannot connect to NFS share on node
Article ID: 400270
Updated On:
Products
Issue/Introduction
Pod produced an error like this:
{"type":"was_message","host":"example_hostname","example_cellName":"ExampleCell01","example_nodeName":"ExampleNode01","example_serverName":"server1","example_sequence":"1748950205640_000000078F856","message":"Encountered an error executing step extractStep in job checkArpExtractJob\norg.springframework.batch.core.UnexpectedJobExecutionException: writeItemsToFile() Exception writing items to file. com.example.fpe.extraction.formatter.TemplateInvoker:: No Output Folder :\/example_hostname\/example_path\/PROD\/AAF\/PROD\/example\/example\Environment
TKGi 1.19.x
TKGi 1.20.x
TKGi 1.21.x
Cause
The error was a function of communication between the POD and the NFS/NAS server. We observed that another share on the same node (mounted by another pod) was able to access the NFS/NAS server; that fact proves this wasn't a global outage. It didn't impact other pods, and didn't impact other nodes. We don't see failures in kubelet or other logs at the timestamp noted in the POD error message:
1748950205640_000000078F856 EPOCH time translates to 2025-06-04T21:23:50 GMT. When checking this timestamp in logging, we saw no errors.
Resolution
The node mount nas_server.example_domain.pvt:/ifs/top_level/folder/prod/be/apps/folder is mounted to the POD on //example_hostname/example_path/ :
Despite this mount being present on the POD, we see the POD present an error indicating a folder within the mount directory is not present:
No Output Folder :\/example_hostname\/example_path\/folder\/folder\/folder\/folder\/folder\
We know that this node has multiple mounts to the same NFS/NAS based on the output of the mount command on the node in question:
worker/########-####-####-####-############:~# mount | grep -i nas_server
nas_server.example_domain.pvt:/ifs/top_level/folder/prod/be/apps/share1 on /var/vcap/data/kubelet/pods/########-####-####-####-############/volumes/kubernetes.io~nfs/example-volume type nfs (rw,relatime,vers=3,rsize=131072,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=###.###.###.###,mountvers=3,mountport=300,mountproto=tcp,local_lock=none,addr=###.###.###.###)
nas_server.example_domain.pvt:/ifs/top_level/folder/prod/be/apps/share1 on /var/vcap/data/kubelet/pods/########-####-####-####-############/volumes/kubernetes.io~nfs/example-volume2 type nfs (rw,relatime,vers=3,rsize=131072,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=###.###.###.###,mountvers=3,mountport=300,mountproto=tcp,local_lock=none,addr=###.###.###.###)
worker/########-####-####-####-############:~#RCA suggests there an issue with this specific NAS share or folder path. We recommend a review on the NFS server logging with the NAS vendor.
Feedback
