HCX - IX/NE appliance error "InvalidGuestLogin: Failed to authenticate with the guest operating system using the supplied credentials"

Products

• VMware HCX
• VMware Cloud on AWS

Issue/Introduction

This resource is to inform about below symptoms and how to recover that:

1. ANY migration workflow failure
2. IX/NE standard redeployment failure
3. Network Stretch Job failure

Symptoms:
HCX Migration workflow will fail to execute and below errors/exceptions can be seen in app-engine logs:

Error 1:

2022-12-22 09:34:51.157 UTC [ReplicationTransferService_SvcThread-15145, Ent: DEFAULT, , TxId: TxId: 8acda383-c4e3-####-####-##########f9] ERROR c.v.h.s.r.jobs.SetupTarget- Job (b98f4467-0309-####-####-##########70) failed with exception java.lang.RuntimeException: 

InterconnectServiceConfigJob failed

Error 2:

2022-12-22 10:53:41.586 UTC [VmotionService_SvcThread-1669, Ent: HybridityAdmin, , TxId: c1fc0607-1355-####-####-##########55] ERROR c.v.v.h.m.common.MigrationUtil- Job (2511a82f-7247-####-####-##########ee) failed with exception System Error. 

Error while logging into Mobility Agent..fault.InvalidLogin.summary

2022-12-22 00:39:15.338 UTC [VmotionService_SvcThread-255582, Ent: HybridityAdmin, , TxId: 4e070f8f-fb36-####-####-##########1d] ERROR c.v.v.h.m.common.MigrationUtil- Job (184868dc-a4b0-####-####-##########75) failed with exception System Error. 

Error while logging into Mobility Agent.. null.

During migration failure, when user try to redeploy IX appliance as an interim recovery using standard method, the redeploy operation will also fail and below error/exceptions can be seen in HCX app-engine log:

Error 3:

2022-12-22 10:50:26.576 UTC [InterconnectService_SvcThread-502341, SM:servicemesh-7f412c6e-2788-####-####-##########76, IX:0fd27cb0-c163-####-####-##########e4, J:799fe681, , TxId: TxId: f7366114-e531-####-####-##########16] WARN c.v.v.h.s.i.GenerateAndPostConfiguration- Failed to post the config using VIX for the appliance f5bedd76-1d6b-####-####-##########d4. Retrying the operation. java.lang.RuntimeException: Failed to upload file to guest @ /tmp/newConfig.proto for vm ServiceMesh-IX-I1

Caused by: com.vmware.vim.binding.vim.fault.InvalidGuestLogin: Failed to authenticate with the guest operating system using the supplied credentials.

Note: The above exception(as highlighted in "Error 3") can also be seen for NE appliances while performing standard redeploy OR enabling new stretch for a given segment/PG.

Location of App Engine log:

• HCX Manager: /common/log/admin/app.log

Resolution

This is fixed in HCX 4.6.0 onwards.

Workaround:
As a potential workaround, one of the step from below can be performed to remediate the "root" password and its validity for the corresponding IX/NE appliances.

STEP 1(UI Based):

• Verify the root password expiration validity for IX or NE appliances.

Go to HCX Manager admin shell >> CCLI >> list >> Locate IX OR NE appliance Id >> go <IX or NE id> >> SSH

admin@vmware-hcx [ ~ ]$ ccli
Welcome to HCX Central CLI

[admin@vmware-hcx] list
|-----------------------------------------------------------------------------------|
| Node                             | Id | Address            | State     | Selected |
|-----------------------------------------------------------------------------------|
| hcx-NE-I1 | 0  | X.X.X.X:9443 | Connected |          |
|-----------------------------------------------------------------------------------|
| hcx-IX-I1 | 1  | X.X.X.X:9443 | Connected |          |
|-----------------------------------------------------------------------------------|

[admin@vmware-hcx] go 1
Switched to node 1.

[admin@vmware-hcx:hcx-IX-I1] ssh
Welcome to HCX Central CLI
root@hcx-IX-I1 [ ~ ]# chage -l root
Last password change                                 : Oct 06, 2022
Password expires                                     : Jan 06, 2022 >>>>>>
Password inactive                                    : never
Account expires                                      : never
Minimum number of days between password change       : 0
Maximum number of days between password change       : 90
Number of days of warning before password expires    : 7

• Refer to HCX Connector/Cloud vCenter Plugin OR Standalone UI where IX/NE appliance creds has been expired.

Go to Interconnect >> Service Mesh >> Appliances >> Click on the IX or NE appliance >> Click "CHANGE PASSWORD"

•  Enter "root" and "admin" credentials in the Interconnect > Multi-Site Service Mesh > Change Password UI and then click UPDATE.

Note: User can choose the custom password for "root" & "admin" account.
Once "UPDATE" option is clicked, a "Task" will be initiated in the backend to update the root/admin credentials and its validity. Please refer SM >> Appliances >> Task section.

Note: The root/admin creds will be changed/modified by HCX manager once SM appliances gets upgraded or redeployed in the future.

STEP 2(CLI Based):

• HCX Manager uses appliance certs to establish SSH connection with its corresponding fleet appliances.

Note: Root password is not used to establish the SSH connection.

• We can run below command in the respective IX or NE root shell via CCLI, to keep the existing password and just to update its validity.

passwd -x -1 root

root@hcx-IX-I1 [ ~ ]# passwd -x -1 root
passwd: password expiry information changed.

root@hcx-IX-I1 [ ~ ]# chage -l root
Last password change                                 : Oct 06, 2022
Password expires                                     : never >>>>>>>>>
Password inactive                                    : never
Account expires                                      : never
Minimum number of days between password change       : 0
Maximum number of days between password change       : -1
Number of days of warning before password expires    : 7

STEP 3: User can also perform "force redeploy" option on IX/NE appliances in a given Service Mesh (SM) to help recover migration and network extension workflow respectively.
Note: The force redeploy option will impact the traffic flow over an existing extended network data path.

Additional Information

IMPORTANT: We have also noticed a similar error/exceptions caught up during Migration, Network Extension & Redeployment workflow, but that has a different underlying cause which requires ONLY "Force Redeploy" option to remediate and fix.
Refer KB HCX - IX/NE appliance configuration failed. Reason: Failed to upload file to guest for more details.

2022-07-18 13:26:57.852 UTC [InterconnectService_SvcThread-48803, SM:servicemesh-217560ff-e780-489b-a428-ae5fa89572a3, IX:dbdb1a18-24df-4d38-9df3-aee896303efb, J:a5e5deba, , TxId: 247d23b9-8430-4219-a40d-915d903dc6a3] INFO c.v.v.h.s.i.GenerateAndPostConfiguration- GenerateAndPostConfig Running in state: 

POST_CONFIG_VIX

2022-07-18 13:26:58.199 UTC [InterconnectService_SvcThread-48803, SM:servicemesh-217560ff-e780-489b-a428-ae5fa89572a3, IX:dbdb1a18-24df-4d38-9df3-aee896303efb, J:a5e5deba, , TxId: 247d23b9-8430-4219-a40d-915d903dc6a3] INFO c.v.v.h.s.i.GenerateAndPostConfiguration- 

About to push config file /common/logs/admin/GW_CONFIG_10.24.242.61-1658150818081.proto to appliance vm-523955

2022-07-18 13:26:58.399 UTC [InterconnectService_SvcThread-48803, SM:servicemesh-217560ff-e780-489b-a428-ae5fa89572a3, IX:dbdb1a18-24df-4d38-9df3-aee896303efb, J:a5e5deba, , TxId: 247d23b9-8430-4219-a40d-915d903dc6a3] ERROR c.v.v.h.s.i.GenerateAndPostConfiguration- 

Update config on cloud gateway failed: File Upload is unsuccessful
java.lang.Exception: File Upload is unsuccessful

Impact/Risks:

• Migration workflow will be affected in the given state of IX appliance.
• The impact is applicable to IX & NE both appliances.
• Upgrade/Redeploy/Resync workflow related to IX/NE appliances won't be serviced.
• New Network Extension services won't be serviced.
• Existing Network Extension datapath will remain unaffected.