VAMI shows Installation Failed: Failed to set vmdir service to maintenance mode, err: trying to update vCenter

Products

• VMware vCenter Server

Issue/Introduction

Trying to update vCenter from 8.0 U3e (8.0.3.00500) to 8.0 U3g (8.0.3.00600) showed the following in VAMI.

vCenter was still accessible through the vSphere Client.

• Installation Failed :
  • Failed to set vmdir service to maintenance mode, err: Failed to set vmdir to maintenance mode
  • The progress bar was at 73% with an option to Resume vCenter update
• Error : "Source vSphere ESX Agent Manager (EAM) upgrade failed to obtain EAM URLs to check against trusted certificates by the System!"
• On checking the vmwPasswordMaxLength & vmwPasswordMinLength using the following command :
  • ldapsearch -o ldif-wrap=no -LLL -h localhost -b "dc=vsphere,dc=local" -s sub "objectclass=vmwPasswordPolicy" -D "cn=Administrator,cn=Users,dc=SSO,dc=DOMAIN" -w 'SSOPWD'
  • The values are different : `ldapsearch` command revealed `vmwPasswordMinLength: 24` and `vmwPasswordMaxLength: 64`
• VMDIR state is NORMAL

Environment

• VMware vCenter Server Appliance 8.0

Cause

• This issue occurs when the machine account password stored in the VMDIRD database does not match the password used by the services to authenticate with the VMware Directory Service.
• The password policy picks up the maximum length value only if the minimum length is greater than 20 characters. The behavior of the password policy is undefined or could result in failure of services when the minimum length value is greater than 20 characters and the maximum length is set to any value. To avoid a potential problem, leave the minimum length set to the default value of 8 characters, or no greater than 20 characters.
• As part of the EAM upgrade pre-checks, an EAM client is created to retrieve all EAM agencies and perform necessary SSL trust checks. This step might not succeed if an EAM client cannot be created because the EAM service is unable to log in to vCenter. This can occur due to a discrepancy between the "vpxd-extension" certificate stored in VECS and the certificate information stored in the vCenter Server Database for the EAM extension.

Resolution

1. Validate the vCenter Domain ID with this command : 
   • /usr/lib/vmware-vmafd/bin/vmafd-cli get-domain-name --server-name localhost
2. Update the vCenter password policy using `ldapmodify` in the below order — set `vmwPasswordMinLength` to 8 and `vmwPasswordMaxLength` to 20

   1. For vmwPasswordMinLength to 8 : 

      ldapmodify -h localhost -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -w 'replace-password' <<EOF
dn: cn=password and lockout policy,dc=vsphere,dc=local
changetype: modify
replace: vmwPasswordMinLength
vmwPasswordMinLength: 8
EOF

   2. For vmwPasswordMaxLength to 20 : 

      ldapmodify -h localhost -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -w 'replace-password' <<EOF
dn: cn=password and lockout policy,dc=vsphere,dc=local
changetype: modify
replace: vmwPasswordMaxLength
vmwPasswordMaxLength: 20
EOF

3. Reset the vCenter Machine Account password   
   1. Check the current vmdir Database state : /usr/lib/vmware-vmafd/bin/dir-cli state get
   2. Expected output if normal: Directory Server State: Normal (3)
   3. If not in Normal state, set it to Normal: /usr/lib/vmware-vmafd/bin/dir-cli state set --state NORMAL
   4. Download and place the reset_machine_pw.sh script from the above article to the vCenter /tmp/ directory and make it executable :
      1. chmod +x reset_machine_pw.sh
   5. Run this command to resolve “bad interpreter” Error: sed -i -e 's/\r$//' reset_machine_pw.sh
   6. Run the reset script : ./reset_machine_pw.sh
   7. Restart All Services : service-control --stop --all && service-control --start --all
4. Update extensions using vCert script from vCert - expired certificate replacement script
   1. Execute following command to update vpxd-extension thumbprint
      

      ./vCert.py --run config/manage_cert/op_manage-vc-ext-thumbprints.yaml
============================================================================================
Please enter a Single Sign-On administrator account [[email protected]]: <-- Press Enter to use the default
Please provide the password for [email protected]: <-- Enter the password for <[email protected]>

      :::

      ------------------------!!! Attention !!!------------------------
Mismatched thumbprints detected.

      Update extension thumbprints? [n]: y   <-- Type 'y' and press Enter
============================================================================================

   2. Restart eam service : 
      service-control --restart eam

5. Re-attempt the vCenter patch process.

1.