Using nfsstat3 to troubleshoot NFS error: Failed to get object: No connection
Article ID: 309934
Updated On:
Products
VMware vSphere ESXi
Issue/Introduction
Symptoms:
- Cannot mount a NFS datastore on an ESX host.
- Cannot reconnect to a NFS datastore if it unmounts.
- Failover from one SP to another on the NFS server takes less than 60 seconds however datastores show as disconnected.
- In vmkernel or messages logs, you see entries similar to:
Oct 30 06:41:02 vmkernel: 129:02:51:50.513 cpu6:49716075)WARNING: NFS: 1830: Failed to get attributes (No connection)
Oct 30 06:41:02 vmkernel: 129:02:51:50.513 cpu6:49716075)FSS: 735: Failed to get object b00f 40 1e2053e9 97f48155 64 f1182d2 820 64000000 f1182d2 a7d5f152 1182d200000064 0 0 0 :No connection
Oct 30 06:41:02 vmkernel: 129:02:51:50.529 cpu1:49716075)FSS: 735: Failed to get object b00f 40 39342bf2 16368dda 64 f10b622 820 64000000 f10b622 8248e439 10b62200000064 0 0 0 :No connection
Oct 30 06:41:02 vmkernel: 129:02:51:50.637 cpu3:49716075)FSS: 735: Failed to get object b00f 40 8af3a074 31d390da 64 f0e57fd 820 64000000 f0e57fd 4b764192 e57fd00000064 0 0 0 :No connection
- In hostd logs, you see entries similar to:
Oct 30 06:40:31 Hostd: [2011-10-30 06:40:31.418 3F6D2B90 verbose 'EnvironmentBrowser'] Skipped inaccessible datastore DatastoreName
Environment
VMware ESXi 4.1.x Installable
VMware ESXi 4.1.x Embedded
VMware vSphere ESXi 5.0
VMware ESX 4.1.x
VMware ESXi 4.1.x Embedded
VMware vSphere ESXi 5.0
VMware ESX 4.1.x
Resolution
The enhanced logging generated by the nfsstat3 service is not enabled by default due to the amount of additional logging which are sent to the logs.
Note: Currently this feature is not implemented in versions beyond ESXi 5.0.
Note: Before enabling this enhanced logging you should follow the troubleshooting steps outlined in Troubleshooting connectivity issues to an NFS datastore (1003967)
To enable the enhanced nfsstat3 logging of this data:
- Log in to the ESXi host as the root user through the console or an SSH session. For more information about SSH, see Tech Support Mode for Emergency Support (1003677)orUsing Tech Support Mode in ESXi 4.1 and ESXi 5.0 (1017910)
- Verify the current setting for the nfsstat3 logging (when disabled the value will be '0'): # esxcfg-advcfg -g /NFS/LogNfsStat3
- Enable the enhanced nfsstat3 logging: # esxcfg-advcfg -s 1 /NFS/LogNfsStat3
- You can verify the value has changed to 1: # esxcfg-advcfg -g /NFS/LogNfsStat3
- When troubleshooting of the issue is complete, disable the enhanced logging with (to prevent the large amounts of log spew): # esxcfg-advcfg -s 0 /NFS/LogNfsStat3
With the enhanced logging enabled, you see these additional information in the system logs. This example is from the /var/log/vmkernel.log file on an ESXi 5.0 host:
<YYYY-MM-DD>T<time> cpu3:373486)NFS: 4544: Received nfsstat3 code 2 for the procedure 3 client 0x41001a501fb0 sock 0x410018d7d170 server's universal address 10.131.8.55.8.1</time>
<YYYY-MM-DD>T<time></time> cpu7:7019844)NFS: 4544: Received nfsstat3 code 70 for the procedure 1 client 0x41002b3045f0 sock 0x410030096010 server's universal address 10.21.150.201.8.1
A status value is returned by the NFS server for every NFS operation sent by the client. The nfsstat3 type is returned with every procedure's results except for the NULL procedure.
A value of NFS3_OK indicates that the call completed successfully. Any other value indicates that some error occurred on the call, as identified by the error code.
The available status codes are:
NFS3_OK = 0,
NFS3ERR_PERM = 1,
NFS3ERR_NOENT = 2,
NFS3ERR_IO = 5,
NFS3ERR_NXIO = 6,
NFS3ERR_ACCES = 13,
NFS3ERR_EXIST = 17,
NFS3ERR_XDEV = 18,
NFS3ERR_NODEV = 19,
NFS3ERR_NOTDIR = 20,
NFS3ERR_ISDIR = 21,
NFS3ERR_INVAL = 22,
NFS3ERR_FBIG = 27,
NFS3ERR_NOSPC = 28,
NFS3ERR_ROFS = 30,
NFS3ERR_MLINK = 31,
NFS3ERR_NAMETOOLONG = 63,
NFS3ERR_NOTEMPTY = 66,
NFS3ERR_DQUOT = 69,
NFS3ERR_STALE = 70,
NFS3ERR_REMOTE = 71,
NFS3ERR_BADHANDLE = 10001,
NFS3ERR_NOT_SYNC = 10002,
NFS3ERR_BAD_COOKIE = 10003,
NFS3ERR_NOTSUPP = 10004,
NFS3ERR_TOOSMALL = 10005,
NFS3ERR_SERVERFAULT = 10006,
NFS3ERR_BADTYPE = 10007,
NFS3ERR_JUKEBOX = 10008
This table gives the description of each defined error:
| Error | Description |
| NFS3_OK | Indicates the call completed successfully. |
| NFS3ERR_PERM | Not owner. The operation was not allowed because the caller is either not a privileged user (root) or not the owner of the target of the operation. |
| NFS3ERR_NOENT | No such file or directory. The file or directory name specified does not exist. |
| NFS3ERR_IO | I/O error. A hard error (for example, a disk error) occurred while processing the requested operation. |
| NFS3ERR_NXIO | I/O error. No such device or address. |
| NFS3ERR_ACCES | Permission denied. The caller does not have the correct permission to perform the requested operation. Contrast this with NFS3ERR_PERM, which restricts itself to owner or privileged user permission failures. |
| NFS3ERR_EXIST | File exists. The file specified already exists. |
| NFS3ERR_XDEV | Attempt to do a cross-device hard link. |
| NFS3ERR_NODEV | No such device. |
| NFS3ERR_NOTDIR | Not a directory. The caller specified a non-directory in a directory operation. |
| NFS3ERR_ISDIR | Is a directory. The caller specified a directory in a non-directory operation. |
| NFS3ERR_INVAL | Invalid argument or unsupported argument for an operation. Two examples are attempting a READLINK on an object other than a symbolic link or attempting to SETATTR a time field on a server that does not support this operation. |
| NFS3ERR_FBIG | File too large. The operation would have caused the file to grow beyond the server's file system to exceed its limit |
| NFS3ERR_NOSPC | No space left on device. The operation would have caused the server's file system to exceed its limit. |
| NFS3ERR_ROFS | Read-only file system. A modifying operation was attempted on a read-only file system. |
| NFS3ERR_MLINK | Too many hard links. |
| NFS3ERR_NAMETOOLONG | The filename in an operation was too long. |
| NFS3ERR_NOTEMPTY | An attempt was made to remove a directory that was not empty. |
| NFS3ERR_DQUOT | Resource (quota) hard limit exceeded. The user's resource limit on the server has been exceeded. |
| NFS3ERR_STALE | Invalid file handle. The file handle given in the arguments was invalid. The file referred to by that file handle no longer exists or access to it has been revoked. |
| NFS3ERR_REMOTE | Too many levels of remote in path. The file handle given in the arguments referred to a file on a non-local file system on the server. |
| NFS3ERR_BADHANDLE | Illegal NFS file handle. The file handle failed internal consistency checks. |
| NFS3ERR_NOT_SYNC | Update synchronization mismatch was detected during a SETATTR operation. |
| NFS3ERR_BAD_COOKIE | READDIR or READDIRPLUS cookie is stale. |
| NFS3ERR_NOTSUPP | Operation is not supported. |
| NFS3ERR_TOOSMALL | Buffer or request is too small. |
| NFS3ERR_SERVERFAULT | An error occurred on the server which does not map to any of the legal NFS version 3 protocol error values. The client should translate this into an appropriate error. UNIX clients may choose to translate this to EIO. |
| NFS3ERR_BADTYPE |
An attempt was made to create an object of a type not supported by the server.
|
| NFS3ERR_JUKEBOX | The server initiated the request, but was not able to complete it in a timely fashion. The client should wait and then try the request with a new RPC transaction ID. For example, this error should be returned from a server that supports hierarchical storage and receives a request to process a file that has been migrated. In this case, the server should start the immigration process and respond to client with this error. |
For more information on these errors, refer to http://www.ietf.org/rfc/rfc1813.txt.
Additional Information
Feedback
Yes
No
Powered by
