Certificate replacement for vCenter fails at re-trust for a Non-VCF vCenter added to the VCF deployed VMware Aria Operations.

Products

• VMware Cloud Foundation 4.x
• VMware vRealize Operations 8.x

Issue/Introduction

When a non-VCF vCenter is added to the VCF-deployed VMware Aria Operations. The certificate replacement for vCenter fails at re-trust with the error - "Could not establish retrust due to: Retrust with VROPS_FQDN failed."

operationsmanager.log - /var/log/vmware/vcf/operationsmanager

2024-11-13T12:22:33.923+0000 DEBUG [vcf_om,f438611ff82f4410,233f] [c.v.v.c.vrops.VROPSSuiteAdapter,om-exec-19] Found suite adapter for VMWARE, perform retrust on resource XXXXXX 2024-11-13T12:22:33.935+0000 DEBUG [vcf_om,f438611ff82f4410,233f] [c.v.e.s.c.c.CertificateUtil,om-exec-19] SSL fingerprint for host XXXXXXX: oldthumbprint 
2024-11-13T12:22:33.936+0000 DEBUG [vcf_om,f438611ff82f4410,233f] [c.v.v.c.vrops.VROPSSuiteAdapter,om-exec-19] Resource Key: VC Adapter Instance for XXXXXXXX CertThumbprint: oldthumbprint requestDto.getId(): newthumbprint 
2024-11-13T12:22:37.160+0000 ERROR [vcf_om,f438611ff82f4410,233f] [c.v.v.c.vrops.VROPSSuiteAdapter,om-exec-19] Failed to update "newthumbprint". com.vmware.ops.api.client.exceptions.ClientSideException: Failed to update "newthumbprint". at com.vmware.ops.api.client.internal.ResponseHandlerImpl.handleResponse(ResponseHandlerImpl.java:99) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:223) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:165) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:140)

Environment

VMware Cloud Foundation 4.x 
VMware vRealize Operations 8.x 

Cause

The issue arises from an incompatibility between vROPS and SDDC. As of version 8.14, vROPS has discontinued support for SHA1 across all services. In contrast, older SDDC versions, including 4.5.1, continue to utilize SHA1 algorithms. Cconsequently, the differing thumbprint algorithms—SHA1 in SDDC and SHA256 in vROPS. 

Resolution

Manual vCenter certificate retrust is performed on the vROPS :

1. Retrieve the vCenter adapter instance :
   GET ​/api​/adapters​/{adapterId}


2. For that adapter, the following API should be used to retrieve the untrusted certificates :
   POST /api​/adapters​/testConnection


3. Then the certificates should be put in the JSON retrieved from the initial GET API call, and that JSON should be sent via the below API to restrust :
   PATCH /api/adapters

The certificate replacement will be completed. However, the error will remain on the SDDC UI as the SDDC remains unaware of the retrust performed on the vROPS end. 

Additional Information

If any task fails due to the vCenter Certificate error, we can suppress the failed task from the SDDC DB (Make sure to assess the failing task before suppressing it)

1. Take a snapshot of SDDC Manager
   
   
2. SSH into SDDC manager as the vcf user and su to the root user
   
   
3. Run the following command to access the operationsmanager DB : 
   psql -h localhost -U postgres -d operationsmanager


4. Run the following select command to check the certificatemanagement schema : 
   select * from certificatemanagement.certificate_operation where operation_status='FAILED';


5. Update the DB to change the statuses from "FAILED" to "SUCCESSFUL" -This includes all failed tasks even old ones :
   update certificatemanagement.certificate_operation set operation_status='SUCCESSFUL' where operation_status='FAILED';


6. Restart the commonsvcs service:
   systemctl restart commonsvcs.service