ESXi upgrade from SDDC Manager fails with error: "VMware Update Manager scan failed.."

Products

• VMware SDDC Manager
• VMware vCenter Server
• VMware Cloud Foundation

Issue/Introduction

• Upgrade of a cluster using baselines fails with the following error:
  
  Message: VMware Update Manager scan failed.

Remediation Message: High: VMware Update Manager scan failed. Refer LCM and VUM log file. Manual intervention needed as upgrade failed during install stage. Check for errors in the lcm log files located on SDDC Manager under /var/log/vmware/vcf/lcm. Please retry the upgrade once the upgrade is available again.
  
  
• vCenter's Lifecycle manager logs shows the following errors:
  
  xxxx-xx-xxTxx:xx:xx.xxxz info vmware-vum-server[29909] [Originator@6876 sub=Libs] [NFC VERBOSE]NfcFile_Send: start
xxxx-xx-xxTxx:xx:xx.xxxz info vmware-vum-server[29909] [Originator@6876 sub=Libs] SSL: syscall error 32: Broken pipe
xxxx-xx-xxTxx:xx:xx.xxxz info vmware-vum-server[29909] [Originator@6876 sub=Libs] [NFC ERROR]NfcNetTcpSetError: Broken pipe
xxxx-xx-xxTxx:xx:xx.xxxz info vmware-vum-server[29909] [Originator@6876 sub=Libs] [NFC ERROR]NfcNetTcpWrite: bWritten: -1. Errno: 32.
xxxx-xx-xxTxx:xx:xx.xxxz info vmware-vum-server[29909] [Originator@6876 sub=Libs] [NFC ERROR]NfcFile_SendMessage: data send failed:
xxxx-xx-xxTxx:xx:xx.xxxz info vmware-vum-server[29909] [Originator@6876 sub=Libs] [NFC ERROR]The operation experienced a network error -- Failed to send data message
..
xxxx-xx-xxTxx:xx:xx.xxxz info vmware-vum-server[29909] [Originator@6876 sub=Libs] [NFC ERROR]NfcSendFile: File send failed for file './docroot/vci/vua/eesx/vua.sh': NFC_NETWORK_ERROR (The operation experienced a network error)
xxxx-xx-xxTxx:xx:xx.xxxz error vmware-vum-server[29909] [Originator@6876 sub=NfcClientWrapper] [nfcClientWrapper 182] Error sending file (error code 3) : The operation experienced a network error
xxxx-xx-xxTxx:xx:xx.xxxz error vmware-vum-server[29909] [Originator@6876 sub=NfcClientWrapper] [nfcClientWrapper 182] [backtrace begin] product: VMware Update Manager, version: <version>, build: build-<build_number>, tag: vmware-vum-server, cpu: x86_64, os: linux, buildType: release
--> backtrace[00] libvmacore.so[0x00531DC5]
..
--> backtrace[15] (no module)
--> [backtrace end]
xxxx-xx-xxTxx:xx:xx.xxxz error vmware-vum-server[29909] [Originator@6876 sub=AgentDeploy] [agentDeploy 262] Cannot copy files to host: <esxi_fqdn>, error: Error sending file (error code 3) : The operation experienced a network error
2025-05-29T13:52:15.862Z info vmware-vum-server[29909] [Originator@6876 sub=Libs] [NFC INFO]Nfc_CloseSessionEx: session=7F914414F0E0
2025-05-29T13:52:15.862Z warning vmware-vum-server[29909] [Originator@6876 sub=Libs] SSL: Unknown SSL Error
2025-05-29T13:52:15.862Z info vmware-vum-server[29909] [Originator@6876 sub=Libs] SSL Error: error:0A00010F:SSL routines::bad length
2025-05-29T13:52:15.862Z info vmware-vum-server[29909] [Originator@6876 sub=Libs] [NFC ERROR]NfcNetTcpSetError: Success
2025-05-29T13:52:15.862Z info vmware-vum-server[29909] [Originator@6876 sub=Libs] [NFC ERROR]NfcNetTcpWrite: bWritten: -1. Errno: 0.
2025-05-29T13:52:15.863Z info vmware-vum-server[29909] [Originator@6876 sub=Libs] [NFC ERROR]NfcSendMessage: NfcNet_Send failed: NFC_NETWORK_ERROR

 

Environment

VMware Cloud Foundation
VMware SDDC Manager
VMware vCenter Server

Cause

This behaviour may indicate a network connectivity problem or a misconfigured firewall between the vCenter and the ESXi hosts.

Resolution

Engage the physical network team to investigate and ensure a stable connection with sufficient bandwidth between the vCenter and the ESXi hosts.