VCF on VxRail - upgrade VxRail task fails on SDDC Manager

Products

• VMware SDDC Manager

Issue/Introduction

While upgrading VxRail in a VCF environment, the task fails on SDDC Manager.

 

Viewing the failed task in the SDDC Manager UI - this is observed - 

• Task - Upgrade DOMAIN - wld_domain_VxRail_fqdn
• Subtask - Upgrade - VXRAIL - wld_domain_vCenter_fqdn
• Description - Upgrade DOMAIN - wld_domain_VxRail_fqdn
• Progress Messages - [vxrail updateUpgradeProgress] VxRail Upgrade returned status as Failure : Please refer to KB00006294 article..
• Error - 
• • Message: [vxrail updateUpgradeProgress] VxRail Upgrade returned status as Failure : Please refer to KB00006294 article..
  • Failure : Please refer to KB00006294 article..
  • Remediation Message: 
  • Reference Token: unique_token
  • Cause: 

 

Open the vSphere Client UI -

- Navigate to the cluster that the upgrade is applicable for

- Identify the host in the cluster that is in maintenance mode and select it

- Select the "Configure" tab on the top row of tabs.

- Select "VxRail", "Updates" on the left hand menu.

- A red banner will be displayed, with this text - "VxRail Update ran into a problem"

- In the "Errors" tab under it - another red banner will be displayed, with this text - "The update identified 2 error(s). Check the lcm-web.log or lcm.log".

- The details of the errors - 

• Failure occurred while running an upgrade for bundle: XXXXX_VxRail-X.X.XXX-Composite-Upgrade-Slim-Package-for-X.X.x.zip. The error message: Failed to login with the management account. Make sure that the username and password are correct.. 
• Upgrade failed. Unexpected failure during upgrade. Check lcm-web.log or lcm.log.

- If "Actions", "Restart Update" is selected - it fails to complete - instead, the workflow throws the same errors.

 

 

Environment

VCF on VxRail 5.x.

 

Cause

This issue occurs if the vCenter service account password is rotated while the upgrade workflow is in progress.

 

The workflow will have been started with the credentials that were valid at the time the workflow was started.

The vCenter service account password will have been subsequently rotated.

The workflow workflow then fails as it is trying to connect to the vCenter with an invalid password.

Resolution

Please contact Dell to assist with addressing the failing task on VxRail.

Once the failing task on VxRail has been cleaned up - restart the workflow on the SDDC Manager.

Additional Information

Confirm the vCenter service account password rotation on the SDDC Manager Ui under Security > Password Management.

The "Last Modified" timestamp should be later than when the upgrade workflow was initiated.