vCenter Services Fail to Start - failed to connect to service. Use service-control command to manage applmgmt service

Products

• VMware vCenter Server

Issue/Introduction

After an outage or power off of the vCenter VM, the vCenter server appliance (VCSA) services fail to start. Many remain in StartPending or Stopped state.

 

While connected to SSH or a remote console of the affected vCenter VM, the following symptoms are observed:

• One of the first messages that appears on screen notes a failure to connect to applmgmt service:
  

  failed to connect to service. Use service-control command to manage applmgmt service.

   

• Applmgmt service is in StartPending or Stopped state when viewing the status of all services:
  

  service-control --status --all

   

• Querying system monitor logs (/var/run/log/vmware/vmon/vmon.log) for applmgmt shows errors similar to the following:
  

  certificate verify failed: IP address mismatch. Certificate is not valid for <vCenter VM IP>

   

• The vCenter VM is using its IP address as its PNID:
  

  /usr/lib/vmware-vmafd/bin/vmafd-cli get-pnid --server-name localhost

Environment

vCenter Server 8.x

Cause

The certificates used by the affected vCenter do not include the vCenter's IP address in the Subject Alternative Name (SAN).

While this is primarily observed in the Machine SSL certificate for the vCenter, this can also affect other certificates.

When prompted to add an IP address (optional), input the vCenter's IP address.

Resolution

Use vCert to renew the Machine SSL certificate and ensure that the vCenter's IP address is included in the Subject Alternative Name (SAN).

1. Start the vCert tool in the affected vCenter:
   • See vCert - Scripted vCenter expired certificate replacement
     
     
     
2. Confirm that the Machine SSL certificate does not contain the vCenter's IP address in SAN:
   1. In the main menu of vCert, go to option 2. View certificate Info
      
      
   2. For the View vCenter Certificates menu, go to option 1. Machine SSL Certificate
      
      
   3. Verify that the certificate is missing the vCenter's IP address under Subject Alternative Names (SAN)
      
      
      
3. Return to the main menu of vCert and enter 3. Manage Certificates
   
   
   
4. For the Manage vCenter Certificates menu, enter option 1. Machine SSL Certificate
   
   
   
5. Choose to replace the Machine SSL certificate according to your intended certificate signer.
   
   1. When prompted with "Enter an IP address (optional)", input the vCenter's IP address.
      
      
      
6. To be thorough, under the Manage vCenter Certificates menu, choose option 2. Solution User certificates and replace the Solution User certificates accordingly
   
   
   
7. Once the above steps complete, restart the affected vCenter's services:
   

   service-control --stop --all && service-control --start --all

   
   
   
8. Confirm that vCenter services are running as expected:
   

   service-control --status --all

Additional Information

Related KB: "Error: Failed to start services in profile ALL" when updating Machine SSL Certificate by certificate-manager when vCenter PNID is IP Address