vCenter Server Appliance (vCSA) Backup Fails After Upgrade - Both Manual and Scheduled Backups Affected

Products

• VMware vCenter Server

Issue/Introduction

After upgrading vCenter Server Appliance (vCSA), both manual and scheduled backups fail with a generic error. This issue can occur even when attempting to back up to a local disk or when using known good credentials.

Environment

- VMware vCenter Server Appliance 7.0 or newer
- vCenter was upgraded
- Previously set up backups, or new backups, failing when run manually or with a schedule
- The following log entries can be observed:

Seen in /var/log/vmware/vpxd/vpxd.log, entries similar to:

2024-07-12T18:24:04.209Z error vpxd[XXXXX] [Originator@XXXX sub=vmomi.soapStub[XXXXX]] Initial service state request failed, disabling pings; /invsvc/vmomi/sdk, <last binding: <<TCP 'XXX.X.X.X : XXXXX'>, <TCP 'XXX.X.X.X : XXXXX'>>>, HTTP Status:400 'Bad Request'

2024-07-12T18:24:15.618Z warning vpxd[XXXXX] [Originator@XXXX sub=Vmomi opID=XXX-XXXX-X] VMOMI activation LRO failed; <<XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX, <TCP 'XXX.X.X.X : XXXX'>, <TCP 'XXX.X.X.X : XXXXX'>>, VpxSettings, vim.option.OptionManager.queryView>, N3Vim5Fault11InvalidName9ExceptionE(Fault cause: vim.fault.InvalidName

Seen in /var/log/vmware/vmdird/vmdird-syslog.log, entries similar to:

2024-07-11T04:06:21.919561+00:00 err vmdird t@XXXXXXXXXXXXX: SASLSessionStep: sasl error (-13)(SASL(-13): authentication failure: client evidence does not match what we calculated. Probably a password error)

2024-07-11T04:06:21.921939+00:00 err vmdird t@XXXXXXXXXXXXX: Bind Request Failed (XXX.XX.XXX.XXX) error 49: Protocol version: 3, Bind DN: "cn=XXXXXXXXXX.XXXXXXX.XXX,ou=Domain Controllers,dc=vsphere,dc=local", Method: SASL

Cause

The mismatch in the machine account password can occur due to the following reasons:

1. Regular password updates: vCenter Server regularly updates its machine account password as a security measure. During the upgrade process, some services may be temporarily offline and miss these password updates.

2. Upgrade process timing: If the upgrade occurs shortly after a scheduled password update, some services might still have the old password while others have already updated to the new one.

3. Upgrade reversion: If you need to revert an update, it can lead to a situation where different components have different versions of the machine account password.

These scenarios result in a discrepancy between the machine account password stored in some vCenter services and the one offered by the vCenter machine user account. This mismatch manifests as LDAP Error Code 49, which interferes with various vCenter operations, including file-based backups.

Resolution

To resolve this issue, follow the steps outlined in the Knowledge Base article:

LDAP Error Code 49 : Reset Machine Account Password of vCenter Server Appliance using Shell Script

This article provides a shell script and instructions to reset the machine account password, which should resolve the LDAP Error Code 49 and restore proper functionality, including the ability to perform backups.

Important: Before proceeding, ensure you have a current snapshot or backup of your vCenter Server Appliance. If you have multiple vCenters in your SSO domain, take offline snapshots of all of them before applying this fix.

After completing the process described in the KB article, attempt a manual backup to verify that the issue is resolved. If successful, you can then re-enable scheduled backups.

Additional Information

- If backup issues persist, check for other potential causes such as network connectivity or firewall rules.