ESXi host intermittently disconnects from vCenter due to InvalidName fault for deprecated setting
search cancel

ESXi host intermittently disconnects from vCenter due to InvalidName fault for deprecated setting

book

Article ID: 426479

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

  • Your ESXi host intermittently shows as disconnected in vCenter Server. When you attempt to reconnect the host, the operation fails initially but eventually succeeds.
  • In the /var/log/hostd.log on the affected ESXi host, you see errors similar to: 
    Failed to read advanced option subtree Disk.SchedNumReqOutstanding: N3Vim5Fault11InvalidName9ExceptionE(Fault cause: vim.fault.InvalidName

    Throw vim.fault.InvalidName
(vim.fault.InvalidName) {
   name = "Disk.SchedNumReqOutstanding",
   msg = "",
}
  • In the ESXi host /var/log/vpxa.log, you see corresponding errors similar to: 
    [VpxLRO] -- ERROR lro-#### -- EsxHostAdvSettings -- vim.option.OptionManager.queryView: vim.fault.InvalidName
  • In the /var/log/vmware/vpxd/vpxd.log on the vCenter Server, the session responsible for the faults may show an older API version similar to: 
    vim.version.version11, official, 6.5

 

Additional symptoms reported:

  • Host shows as disconnected while working in vCenter
  • Clicking "Reconnect" produces the error: "Cannot contact the specified host. The host may not be available on the network, a network configuration problem may exist, or the management services on this host may not be responding."
  • Host eventually reconnects successfully
  • Similar behavior observed on multiple ESXi hosts
  • VM backup failures due to VMs appearing disconnected |

Environment

  • VMware vCenter Server 7.x or 8.x
  • VMware ESXi 7.x or 8.x
  • Environments upgraded from ESXi 6.x, managed by:
    • vSphere Lifecycle Manager (vLCM) / VMware Update Manager (VUM) with legacy baseline references, or
    • Managed by third-party tools using older vSphere API versions

Cause

The Disk.SchedNumReqOutstanding advanced setting was deprecated in vSphere 5.5 and removed entirely in ESXi 7.0. However, stale references to this setting can persist in the environment from two potential sources:

  • The vSphere Lifecycle Manager (vLCM) or VMware Update Manager (VUM) database, which may retain references from older baseline configurations.
  • Third-party management, monitoring, or storage integration tools that connect to vCenter using older vSphere API bindings (such as the vSphere 6.5 API). These tools may periodically query ESXi advanced settings that no longer exist. Because these tools operate at the vCenter level, they can affect hosts across all clusters managed by that vCenter, even hosts that are not directly associated with the tool's intended scope.

When vLCM/VUM or an external tool queries this setting on an ESXi 7.x or 8.x host, the host returns an InvalidName fault because the setting no longer exists. These repeated faults destabilize the communication between the hostd and vpxa services, causing the host to intermittently appear disconnected in vCenter.

Resolution

Option 1: Reset the vLCM/VUM database

If the faults are caused by stale references in the vSphere Lifecycle Manager or VMware Update Manager database:

  1. Connect to the vCenter Server Appliance using SSH or the VAMI console.
  2. Create a powered-off snapshot of the vCenter Server Appliance before proceeding.
  3. Reset the vSphere Lifecycle Manager (vLCM) / VMware Update Manager (VUM) database by following the steps in Resetting the VMware Update Manager Database (316581).
  4. If you use Broadcom download token URLs for vLCM patch downloads, re-add these URLs after the database reset. The reset removes custom download source configurations.
  5. Monitor the environment over the following days to confirm the ESXi host disconnection events do not recur.

Option 2: Identify and update the third-party tool

If the faults persist after the vLCM/VUM database reset, or if the faults are attributed to a non-VMware service account, a third-party tool is likely the source. Follow these steps to identify it.

Key identifiers

Before starting, understand the three identifiers used to trace the fault to its source:

  • opID (Operation ID): A short hex string (e.g., 1234abcd) that tags a single API operation as it flows through the vpxd, vpxa, and hostd services. Use it to correlate the same operation across all three logs.
  • Session ID: A UUID (e.g., 1234abcd-1234-abcd-1234-1234abcdef12) that identifies the authenticated vCenter SDK session. Every API call in vpxd is tagged with the session ID of the caller. In the /var/log/vmware/vpxd.log, the session ID appears in two places:
    • In VMOMI activation LRO failed lines: <<SESSION_ID, <TCP ...
    • In VpxLRO ERROR lines: SESSION_ID(USER_PRINCIPAL_ID)
  • User principal ID: A UUID in parentheses after the session ID in VpxLRO lines (e.g., 1234abcd-1234-abcd-1234-1234abcdef12). This is the SSO identity tied to the session. It is not needed for this investigation but appears alongside the session ID.

Step 1: Find the fault in vpxd.log and identify the session

On the vCenter Server, open /var/log/vmware/vpxd.log and search for InvalidName. Look for VMOMI activation LRO failed warning lines that reference vim.option.OptionManager.queryView. For example:

YYYY-MM-DDTHH:mm:ss.###Z warning vpxd[06885] [Originator@6876 sub=Vmomi opID=1234abcd]
  VMOMI activation LRO failed;
  <<1234abcd-1234-abcd-1234-1234abcdef12, <TCP '127.0.0.1 : 8085'>, ...>>,
  EsxHostAdvSettings-577538, vim.option.OptionManager.queryView,
  <vim.version.version11, official, 6.5>, ...

From this line, extract:

Field Where to find it Example value
opID In the log header: opID=1234abcd 1234abcd
Session ID Immediately after << at the start of the details block 1234abcd-1234-abcd-1234-1234abcdef12
API version After the managed object and method, in angle brackets vim.version.version11, official, 6.5

A session using an old API version such as vim.version.version11 (vSphere 6.5) indicates a tool that has not been updated for current ESXi versions. If the API version is v8_0_#_# or similar, the fault is being propagated by a vCenter internal session rather than originating from an external tool — in that case, look for other sessions in the same log that also hit this fault with an older API version.

Note: If the fault entries only show the internal vCenter session (v8.0 API), the external tool's session that triggered the query may also be present in the same log with a different session ID and an older API version. Search for all InvalidName lines and compare the session IDs and API versions across them.

Step 2: Find the session login event

Search the same /var/log/vmware/vpxd.log (and earlier rolled vpxd log files, including *.gz files) for the session ID identified in Step 1. Look for the SessionManager.login line:

YYYY-MM-DDTHH:mm:ss.###Z info vpxd[06375] [Originator@6876 sub=vpxLro opID=1234abcd]
  [VpxLRO] -- BEGIN lro-3010991 -- SessionManager -- vim.SessionManager.login --
  1234abcd-1234-abcd-1234-1234abcdef12

Immediately after this line, the SsoClient entry reveals the authenticated user:

YYYY-MM-DDTHH:mm:ss.###Z info vpxd[06375] [Originator@6876 sub=SsoClient opID=1234abcd]
  Successfully acquired token: SamlToken [subject={Name: VENDOR_USER_ACCOUNT; Domain:VSPHERE.LOCAL}, ...]

Note the username (VENDOR_USER_ACCOUNT) and the timestamp of the login (YYYY-MM-DDTHH:mm:ss).

If the login event is not found in the current /var/log/vmware/vpxd/vpxd.log file, it may be in an older rolled log. vpxd logs can be named vpxd-###.log (or vpxd-###.log.gz for compressed older files). Search them in descending order by number. Use zgrep for *.gz files:

zgrep "SessionManager.*login.*1234abcd-1234-abcd-1234-1234abcdef12" /var/log/vmware/vpxd/vpxd-*.log.gz

Step 3: Match the login timestamp to journalctl to find the source IP

Open the vCenter authentication log. In a log bundle, this is commands/journalctl_-b--0.txt. On a live system, use journalctl -b -0. Search for the username from Step 2 together with UserLoginSessionEvent:

journalctl -b -0 |grep "VENDOR_USER_ACCOUNT" | grep "UserLoginSessionEvent"

Find the entry whose timestamp matches (or is within 1 second of) the SessionManager.login timestamp from Step 2:

MMM DD HH:mm:ss <some server fqdn> vpxd[6177]:
  [vim.event.UserLoginSessionEvent] [info] [VSPHERE.LOCAL\VENDOR_USER_ACCOUNT]
  [User VSPHERE.LOCAL\VENDOR_USER_ACCOUNT@<ip address>> logged in as Java/11.0.29]

This line provides:

Field Where to find it Example value
Source IP After @ in the user identifier ###.###.###.###
User agent After logged in as Java/11.0.29

The UserLogoutSessionEvent entries for the same user are also useful — they show the login cadence and the number of API invocations per session, which helps characterize the tool's behavior.

Step 4: Identify and update the tool

Once the source IP is identified, determine what application is running on that system. Common culprits include backup software, storage array integration plugins, and monitoring tools that were originally deployed against a vSphere 6.x environment and never updated.

Update the third-party tool to a version that uses current vSphere API bindings. Contact the tool's vendor for guidance on the update process if needed. If you're already on the latest version, the vendor should be requested t update their tool to use the corrrect, non-deprecated or removed API calls

After the update, and after the tool has been running on the new version for at least 24 hours, check /var/log/hostd.log on previously affected hosts to confirm the InvalidName faults have stopped.

If the faults continue after updating, contact the tool's vendor and report the issue. The tool is querying a deprecated ESXi setting that was removed in ESXi 7.0, and the vendor will need to update their software to remove that query.

Additional Information

Powered by Wolken Software