VM deployment from content library is taking longer than usual causing impact on the deployment
search cancel

VM deployment from content library is taking longer than usual causing impact on the deployment

book

Article ID: 391010

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

  • Deployment of virtual machines using content library templates takes upwards of 20 minutes at times. 
  • This behavior is consistently observed when the datastore hosting the content library template differs from the target datastore designated for the deployed virtual machine.
  • If source and destination datastores are same, deployments proceed within expected timeframes.
  • Observing slow file transfer rate in the /var/log/vmware/content-library/cls.log

DEBUG    | c5b16059-####-####-####-############| transferService-pool-7-thread-7380 | TransferItemImpl | Session a4c2a346-####-####-####-############, Item <VMTEMPLATENAME>.vmdk: Transfer summary: rate=25.03 MB/s, size=62914560, time=2514 ms, status=DONE, type=DISK, source=ds:///vmfs/volumes/vsan:########//contentlib-########/VMTEMPLATENAME_##########.vmdk?serverId=########, target=ds:///vmfs/volumes/vsan:########/VMNAME.vmdk

  • ESXi host log: /var/run/log/hostd.log

    • NFC VERBOSE logging

      info hostd[2107031] [Originator@6876 sub=Libs opID=ebf7####-####-####-####-############-25-72-ef12 user=vpxuser:<vSpehre_SSO_Domain>\vpxd-extension-############] [NFC VERBOSE]NfcFile_Send: about to send 261656 bytes from file
      info hostd[2107031] [Originator@6876 sub=Libs opID=ebf7####-####-####-####-############-25-72-ef12 user=vpxuser:<vSpehre_SSO_Domain>\vpxd-extension-############] [NFC VERBOSE]NfcFile_Send: about to send 261656 bytes from file 

    • Fluctuating NFC buffer size

      info hostd[2107372] [Originator@6876 sub=Libs opID=m2pqqej9-########-auto-yzi6-h5:########-8f-01-602e user=vpxuser:domain\username] NfcNet_OptimizeSocketBuffer: Setting buffer size using RTT=16 ms BANDWIDTH=134000 bytes/ms BUFFER_SIZE=2167KB
      info hostd[2110754] [Originator@6876 sub=Libs] NfcNet_OptimizeSocketBuffer: Setting buffer size using RTT=2 ms BANDWIDTH=134000 bytes/ms BUFFER_SIZE=286KB

Environment

VMware vCenter Server 7.x

VMware ESXi Server 7.x

Cause

  • In VDDK versions lower than 8.0, backup vendors had an option to switch the NFC log levels for debugging and troubleshooting purposes.
  • When the source and destination datastores are not the same, the NFC transfer operation is employed for file transfer.
  • Enabling verbose NFC logging can significantly impact on performance of the file transfer operation causing the increased virtual machine deployment time.

Resolution

  • With ESXi 7.0.3, the log-level of hostd needs to be updated using the below command:
    • Command to check the log level status:

vim-cmd hostsvc/advopt/view Config.HostAgent.log.level

    • If the hostd log level is verbose, set the log level to info:

vim-cmd hostsvc/advopt/update config.HostAgent.log.level string info

    • Restart hostd service:

/etc/init.d/hostd restart

  • Investigate the Backup server for GFLAGS, for increased logging level for NFC related logs. These flags are enabled by the Backup server for debugging purpose.

     

    • For Example, in Cohesity backup, the command to check the GFLAGS is: iris_cli cluster ls-gflags vervose=1

  • These GFLAGs need to be disabled by backup vendor, in case if it is not needed.
  • Once the GFLAGs are disabled, restart of hostd service is required on all the hosts in the cluster.

/etc/init.d/hostd restart

Additional Information

  •  With the previous VDDK versions, backup vendors had an option to switch the NFC log levels for debugging purpose.
  • In VDDK 8.0 updates, the NFC server logging level will no longer configurable from the VDDK client configuration file. This was originally for debugging but will be discontinued for security reasons.
Powered by Wolken Software