Adding ESXi 8.0 to vCenter fails with "External command failed" or "An error occurred while communicating with the remote host"
Article ID: 443046
Updated On:
Products
Issue/Introduction
Adding ESXi 8.0 to vCenter fails with "External command failed" or "An error occurred while communicating with the remote host"
When attempting to add a standalone ESXi 8.0 (or 8.0 U3) host to vCenter Server, the task fails at 80% with one of the following UI errors:
Error 1:
A general system error occurred: External command failed
Failed to configure the VIM account on the host
Error 2:
An error occurred while communicating with the remote host.
In the vCenter vpxd.log, you observe the following sequence:
warning vpxd [...] Terminating invocation; [...] method: selectCapabilities
info vpxd [...] SOAP request returned HTTP failure; [...] method: retrieveTaskManager; code: 500(Internal Server Error); fault: (vpxapi.fault.NotAuthenticated)
error vpxd [...] [VpxLRO] Unexpected Exception: N5Vmomi5Fault17HostCommunication9ExceptionE(Fault cause: vmodl.fault.HostCommunication
You may also see warnings indicating policies already exist:
warning vpxd [...] [CreateOrUpdate] Policy with local user name vpxuser already exists on host <hostname>. Exception: N4Vpxd7Clients22AlreadyExistsExceptionE
In the ESXi hostd.log, you observe the host failing to create the vpxuser account :
Er(163) Hostd [...] SysCmd [/usr/lib/vmware/auth/bin/adduser ++group=hostd-tmp,mem=30 -H -h / -s /bin/sh -g VMware Workstation administration account -D vpxuser] failed with exit code [1] and output:
Er(163) Hostd [...] --> adduser: group 'vpxuser' in use
In(166) Hostd [...] Throw vmodl.fault.SystemError
Environment
vCenter Server
Cause
This issue is caused by a two part failure in the ESXi host's management and authentication layers, typically occurring if a previous attempt to add the host timed out, failed, or was improperly cleaned up.
Orphaned vpxuser OS Group:
If the vpxuser account was previously removed manually from /etc/passwd but the group was left behind in /etc/group, the ESXi hostd service will fail to recreate the account during the vCenter handshake, resulting in the External command failed system error.
Stale Trusted Infrastructure (WCP/OAuth) Policies:
In vSphere 8.x, vpxa and Envoy use JSON Web Tokens (JWT) and WCP policies to authenticate vCenter traffic. If the OS-level vpxuser account is altered or recreated, the ESXi host may still cache the old authentication policies. vCenter assumes these policies are in the desired state, but the host rejects the incoming token mismatch with a vpxapi.fault.NotAuthenticated fault, dropping the connection.
Resolution
To resolve this issue, you must cleanly remove the orphaned vpxuser group from the ESXi host and regenerate the host certificates to force a flush of the stale Trusted Infrastructure policies.
Step 1: Clean up the orphaned vpxuser group :
Connect to the affected ESXi host via SSH as root.
Ensure the actual vpxuser account is removed from the system:
In ESXi versions prior to 8.0 Update 3
Connect to the host(s) via SSH and login using the root account
Stop the vpxa service:# /etc/init.d/vpxa stop
Restart the hostd service:# /etc/init.d/hostd stop && /etc/init.d/hostd start
Run the following command to delete the existing vpxuser account:# esxcli system account remove -i vpxuser
Start the vpxa service:# /etc/init.d/vpxa start
In ESXi 8.0 Update 3 and later versions
Connect to the host(s) via SSH and login with root
- Stop the vpxa service:
# /etc/init.d/vpxa stop
- Create backup copies of the files :
# cp /etc/passwd /etc/passwd.bak# cp /etc/shadow /etc/shadow.bak# cp /etc/group /etc/group.bak
- Remove the orphaned vpxuser group entry using sed:
# sed -i '/^vpxuser:/d' /etc/group# sed -i '/^vpxuser:/d' /etc/passwd# sed -i '/^vpxuser:/d' /etc/shadow
Step 2: Regenerate ESXi Certificates :
Regenerating the self-signed certificates will force vCenter to treat the host as a new cryptographic entity, bypassing the stale WCP/OAuth policies.
- Navigate to the SSL directory:
# cd /etc/vmware/ssl/
- Backup the existing certificate and key:
# mv rui.crt rui.crt.bak# mv rui.key rui.key.bak
- Generate new certificates:
# /sbin/generate-certificates
Step 3: Restart Management Agents :
- Restart the host daemon, vCenter agent, and reverse proxy to ingest the clean configuration and new certificates.
# /etc/init.d/hostd restart# /etc/init.d/vpxa restart# /etc/init.d/rhttpproxy restart
Step 4: Reconnect the Host
Log into the vSphere Client.
Attempt to add the standalone ESXi host again.
When prompted, accept the new certificate thumbprint. The host will now successfully complete the handshake and configure the VIM account.
Feedback
