Unable to migrate VMs using HCX with message "Cloud Resources are unavailable"
search cancel

Unable to migrate VMs using HCX with message "Cloud Resources are unavailable"

book

Article ID: 328977

calendar_today

Updated On:

Products

VMware HCX

Issue/Introduction

  • Cannot migrate VMs, "Cloud Resources are unavailable"
  • Cannot protect VMs using HCX DR: "Cloud Resources are unavailable"

  • Cannot recover VMs with HCX DR: "Replication destination container is missing, unable to protect or recover VM"

Environment

VMware HCX

Cause

This issue is due to an incorrect username or password used for site pairing.

Example 1
Logs: common/logs/admin/web.log

<time stamp> UTC [https-jsse-nio-8443-exec-4, , , TxId: ] ERROR c.v.v.h.a.HybridityAuthenticationEntryPoint- AuthenticationEntryPoint - got AuthenticationException
org.springframework.security.authentication.BadCredentialsException: Error validating user [email protected]: Status code: 500, Reason: Internal Server Error<?xml version='1.0' encoding='UTF-8'?>
<S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/"><S:Body><S:Fault xmlns:ns4="http://www.w3.org/2003/05/soap-envelope"><faultcode xmlns:ns0="http://docs.oasis-open.org/ws-sx/ws-trust/200512">ns0:Fa
iledAuthentication</faultcode><faultstring>Password of the user logging on is expired. :: Password of the user logging on is expired. :: User account expired: {Name: ########, Domain: vsphere.local}</
faultstring></S:Fault></S:Body></S:Envelope>

Example 2
Logs: common/logs/admin/web.log

<time stamp>[https-jsse-nio-8443-exec-7, Ent: HybridityAdmin, , TxId: TxId: ######-####-####-####-##########] WARN  c.v.vca.hybridity.util.NSPRestClient- Login to cloud https://######.example.com failed, with status 401:  <time stamp> [https-jsse-nio-8443-exec-7, Ent: HybridityAdmin, , TxId: TxId: ######-####-####-####-########## WARN  c.v.vca.hybridity.util.NSPRestClient- Invalid credentials logging into remote endpoint: https://######.example.com, userName: [email protected] <time stamp> [https-jsse-nio-8443-exec-7, Ent: HybridityAdmin, , TxId: TxId: ######-####-####-####-##########] WARN  c.v.vca.hybridity.util.NSPRestClient- Login failure to 20230126221226503-01159884-####-####-####-############ <time stamp> [https-jsse-nio-8443-exec-7, Ent: HybridityAdmin, , TxId: TxId: ######-####-####-####-############] WARN  c.v.v.h.inventory.InventoryAdapter- Can not collect resources from the cloud: https://######.example.com
java.lang.RuntimeException: Invalid credentials

Resolution

Option 1

  • Access HCX Manager on port 9443
  • Re-enter vCenter credentials for @vsphere.local user.
  • Restart the services - How to restart HCX Manager services. 
  • Otherwise, Restart services (source and cloud): Use "admin" credentials to SSH into the HCX Connector or Cloud Manager and change the user to "root".
  • Restart the services as shown below.
    • # systemctl restart app-engine
    • # systemctl restart web-engine
    • # systemctl restart appliance-management
  • Update the credentials for the site pair. 


Option 2 - For HCX 4.10 and earlier versions

    1. Login to HCX API Explorer: https://<HCX connector-url or IP>/hybridity/docs/index.html
    2. From the "Choose Product" drop-down, select "HCX connector".
    3. ​​​​​​​​​​​First, get the authorization code by using the below API:
      1. Click on Platform ---> /hybridity/API/sessions
      2. Click "Try it out"
      3. Enter HCX Connector username and password and click Execute
      4. Response code should be 200 and at the end you can find the x-hm-authorization token, copy the code for later use.
    4. Now use this authorization code, to authenticate cloud registration.
      1. Click on Cloud Registration and Authorize this API calls:
      2. Enter the value from point 3.IV
      3. Click "Try it out" under "POST /hybridity/api/cloudConfigs" and enter the values of the Cloud HCX manager IP address, Username and password. Click Execute.
      4. Verify that a response 200 is returned:


Note:

If there are no underlying issues, site pairing via the API should successfully re-establish the connection. It has been observed that due to stale entries in the 'RemotingOutbox' collection due to the site pairing being down, the site pairing via the API may not help. In such cases, it is necessary to check the status of the 'RemotingOutbox' in the HCX database and clear any outdated entries. Please contact Broadcom Support for more information on this:  Contact Broadcom support

Additional Information

Powered by Wolken Software