HCX - Failure to deploy IX or NE appliance with "InvalidGuestLogin" error

Products

• VMware HCX

Issue/Introduction

Identify a condition that may be blocking the deployment of HCX fleet appliance VMs and provide a procedure to clear it.

Symptoms:
Deployment, redeploy, or upgrade of HCX fleet appliances may fail with the following error:

Redeploy of Interconnect Appliance failed. Interconnect Service Workflow InterconnectRedeploy failed. Error:["Interconnect Service Workflow GenerateAndPostConfig failed. Error: [\"Failed to upload file to guest @ /tmp/newConfig.proto for vm.

Cause: Failed to authenticate with the guest operating system using the supplied credentials.\"]"]

UpgradeAppliance Failed. Reason:Interconnect Service Workflow InterconnectRedeploy failed. Error: ["Redeploy of Interconnect Appliance failed. Interconnect Service Workflow InterconnectRedeploy failed. Error:["Interconnect Service Workflow GenerateAndPostConfig failed. Error: [\"Failed to upload file to guest @ /tmp/newConfig.proto for vm.

Cause: Failed to authenticate with the guest operating system using the supplied credentials.\"]"]

Cause

The failure in the deployment process is due to the vpxa and hostd logging level set to "trivia" on the ESXi hosts.
The increased load on the hostd service to generate the detailed log messages introduces a delay in getting the initial configuration applied to the VM which leads to the race condition that causes the deployment to fail.

Resolution

Reduce the logging level on the ESXi hosts in the deployment cluster.

• In the vCenter UI, go to ESXi host ---> Configure ---> System ---> Advanced System Settings
• Search for the following logging sources:

Vpx.Vpxa.config.log.level 

config.HostAgent.log.level 

• Change the value from "trivia" to "info" for both

• Restart the ESXi management agents using below CLI:

/etc/init.d/hostd restart
/etc/init.d/vpxa restart

• Deploy new Service Mesh or redeploy existing appliances

Workaround:
None.

Additional Information

Impact/Risks:
New Service Mesh configuration will not be able to deploy any fleet appliances so Migration and Network Extension services will be unavailable.
Appliances for an existing Service Mesh may get disrupted during redeployment or upgrade.