Inventory sync fails for VCF Operations for Logs from VCF Fleet Management – Fails at Stage 1 with Error Code: LCMVRLISYSTEM45018

Products

• VCF Operations

Issue/Introduction

Inventory sync for VCF Operations for Logs initiated from VCF Fleet Management fails at Stage 1 with error code LCMVRLISYSTEM45018. The task does not progress and reports an IO-related failure, even though the VCF Operations for Logs instance is up and running.

Error Message:

Error Code: LCMVRLISYSTEM45018
IO error , Check Operations-logs instance is up and running.
IO error , Check Operations-logs instance is up and running.

Log Snippet in /var/log/vrlcm/vmware_vrlcm.log:

INFO vrlcm[######] [pool-3-thread-31] [c.v.v.l.u.CustomTrustManager]  -- Fetching certificate chain from https://<Ops_for_Logs_FQDN_IP>/api/v2/certificate-management/certificate-bundle
INFO vrlcm[######] [pool-3-thread-31] [c.v.v.l.u.NdcHelper]  -- Successfully fetched JWT token from https://<Ops_for_Logs_FQDN_IP>/api/v2/certificate-management/certificate-bundle
INFO vrlcm[######] [pool-3-thread-31] [c.v.v.l.u.NdcHelper]  -- JWT Headers {"kid":"<CERTIFICATE_THUMBPRINT>","alg":"RS512"}
INFO vrlcm[######] [pool-3-thread-31] [c.v.v.l.u.NdcHelper]  -- Check whether certificate exist with thumbprint - <CERTIFICATE_THUMBPRINT>
ERROR vrlcm[######] [pool-3-thread-31] [c.v.v.l.u.NdcHelper]  -- No certificate found in trust store for thumbprint <CERTIFICATE_THUMBPRINT>
ERROR vrlcm[######] [pool-3-thread-31] [c.v.v.l.u.CustomTrustManager]  -- Failed to trust certificate
java.lang.RuntimeException: No certificate found in trust store for kid <CERTIFICATE_THUMBPRINT>

Environment

VCF Operations 9.0.x
VCF Fleet Management 9.0.x
VCF Operations for Logs 9.0.x

Cause

The failure occurs because the certificate presented by the VCF Operations for Logs endpoint is not trusted by VCF Fleet Management. During inventory sync, VCF Fleet Management attempts to fetch and validate the certificate bundle, but the certificate thumbprint referenced in the JWT header is missing from the trust store.

Resolution

To resolve the issue and resume the inventory sync, follow the steps below:

 

Trust the Certificate via Fleet Management API

• Log in to the VCF Operations product UI as admin user. (https://<VCF_OPS_FQDN_IP>/ui)
• Navigate to Developer Center > API & SDKs > Fleet Management API
  
  
  
  
• In the Swagger UI enter the Authorize key. Refer to ' How to Authorize VCF Operations Fleet Management API '
• Scroll to Certificate Management Controller API
• Use the following POST API call, /lcm/certificate-management/api/certificate-trust
  
  
  
  
• Click "Try it Out".
  
  
  
• In the endpoint field, enter the FQDN mentioned in the error logs (i.e., <Ops_for_Logs_FQDN_IP>).
• Click Execute
  
  
  
  
• Ensure the HTTP response code is 200, indicating success.

Retry the Inventory Sync Task

• Log in to the VCF Operations product UI as admin user. (https://<VCF_OPS_FQDN_IP>/ui)
• Navigate to Fleet Management > Lifecycle > Tasks
  
  
  
  
• Select the previously failed inventory sync task.
• Click on Retry.
• The task should now complete successfully.