Backup/Restore Performance Issues over VDDK NBD Transport Mode

Products

• VMware vSphere ESXi
• VMware vCenter Server

Issue/Introduction

Customers may experience slow backup or restore performance when using the VDDK NBD (Network Block Device) transport mode.

Environment

VMware vCenter Server 8.x

VMware vCenter Server 9.x

VMware vCenter ESXi 8.x

VMware vCenter ESXi 9.x

 

Cause

This issue can occur when the NFC (Network File Copy) logging level on the ESXi host is configured to a verbose setting such as DEBUG or TRIVIA. Excessive NFC logging can negatively impact data transfer performance during backup and restore operations.

The following entries may be observed in the hostd.log file:

xxxx-xx-xx:xx:xx.xx In(166) Hostd[#######]: [Originator@6876 sub=Libs opID=nbdmode-##################] [NFC VERBOSE]NfcFssrvrDecompressSkipZ: Zeroing ########
xxxx-xx-xx:xx:xx.xx In(166) Hostd[#######]: [Originator@6876 sub=Libs opID=nbdmode-##################] [NFC VERBOSE]NfcFssrvrDecompressSkipZ: Copying ########
xxxx-xx-xx:xx:xx.xx In(166) Hostd[#######]: [Originator@6876 sub=Libs opID=nbdmode-##################] [NFC VERBOSE]NfcFssrvrDecompressSkipZ: Zeroing tail #######
xxxx-xx-xx:xx:xx.xx In(166) Hostd[#######]: [Originator@6876 sub=Libs opID=nbdmode-##################] [NFC VERBOSE]NfcAioServerProcessMain: Exiting
xxxx-xx-xx:xx:xx.xx In(166) Hostd[#######]: [Originator@6876 sub=Libs opID=nbdmode-##################] [NFC VERBOSE]NfcAioServerProcessMain: Exiting processing loop
xxxx-xx-xx:xx:xx.xx In(166) Hostd[#######]: [Originator@6876 sub=Libs] [NFC VERBOSE]NfcAioIoCB: Schedule main processing from IO callback

 

Resolution

Revert the NFC logging configuration on the affected ESXi host(s) to the default logging level and then retest the backup or restore operation.

1. Dump the current hostd block.
configstorecli config current get -c esx -g services -k hostd -outfile /tmp/hostd.json

Note: Create a backup copy of /tmp/hostd.json before making any modifications.

2. Edit the hostd.json file to make the necesssay log level changes and am using vi command from the ESXi host.

#  vi /tmp/hostd.json — type /nfcsvc<Enter>. You will see:
#       "nfcsvc": {
#          "log_level": "DEBUG",          <-- change this value
#          "max_memory": 100663296,
#          "max_stream_memory": 35651584
#       },
#    Change ("DEBUG" or "VERBOSE") to "INFO", then save th file using :wq command

3. Apply — NO --overwrite flag.
configstorecli config current set -c esx -g services -k hostd -j /tmp/hostd.json

4. Restart hostd so the NFC library re-reads the level.
/etc/init.d/hostd restart

5. Verify:
configstorecli config current get -c esx -g services -k hostd -outfile /tmp/hostd.json
grep -A3 '"nfcsvc"' /tmp/hostd.json

 

Additional Information

If you require further assistance, please contact the Broadcom VCF SDK/API Support team by following the KB: Creating API and SDK Technical Support Cases for Broadcom VCF Environments